Introduce Multi-Threading

[damoclark]

Hi @rfjakob

Thanks for sharing cshatag. PR implements goroutines to perform the hashing and updating of the xattrs.

I've made the following changes:

1. Updated the man page including the new -jobs flag for specifying the number of goroutines to execute.
2. Implemented a channel for the walker (or the iteration of the command arguments) to pass fn paths to the goroutines to process
3. Updated the stats struct to use uint32 so they can be incremented atomically in the goroutines
4. Introduced a signal handler so that the process can be terminated gracefully, and ensure that either both or neither xattrs are set on a file
5. Introduced module northbright/iocopy that allows the io.copy call to be interrupted by the signal handler using Context
6. Updated the test.sh script to handle macOS/BSD touch command syntax

I'm not new to system programming, but I am new to Go. Please run a careful eye over my code.

If you would like to merge, please make any changes/improvements you feel necessary.

Hope this PR proves useful.

Cheers,
Damien.

[rfjakob]

Hi, thanks for the PR!

I don't think we need the context thing. Bang you're dead is fine.

Benchmark results would be nice.

A test would be nice.

[rfjakob]

Please use 64-bit values, and use the new type atomic.Uint64 as ( https://pkg.go.dev/sync/atomic#Uint64 )

[rfjakob]

No new globals, please

[rfjakob]

Also I don't think we need the context at all

[rfjakob]

Please make this a separate commit that explains the problem you are solving here

[rfjakob]

I don't think we need any of this. Bang you're dead works just fine for cshatag. Nothing bad happens if we write only one xattr.

[rfjakob]

uint?

[rfjakob]

Please handle jobs=0. You can just exit with an error like "make -j0" does:

$ make -j0
make: the '-j' option requires a positive integer argument

[rfjakob]

Please don't delete the readme :)

[rfjakob]

Not sure about the queue depth here. Does it get faster with a deeper queue?

Otherwise I would just do

queue = make(chan string)

[rfjakob]

Also, there's a race condition somewhere. I did not immediately spot where.

With

fmt.Printf("stats.total=%d\n", stats.total)

before exit and running against the extracted https://cdn.kernel.org/pub/linux/kernel/v3.0/linux-3.0.tar.gz :

6 jakob@brikett:~/code/cshatag$ ./cshatag -recursive -qq -jobs 2 /tmp/linux-3.0
stats.total=36768
6 jakob@brikett:~/code/cshatag$ 
6 jakob@brikett:~/code/cshatag$ ./cshatag -recursive -qq -jobs 2 /tmp/linux-3.0
stats.total=36774
6 jakob@brikett:~/code/cshatag$ ./cshatag -recursive -qq -jobs 2 /tmp/linux-3.0
stats.total=36775
6 jakob@brikett:~/code/cshatag$ ./cshatag -recursive -qq -jobs 1 /tmp/linux-3.0
stats.total=36782
0 jakob@brikett:~/code/cshatag$ ./cshatag -recursive -qq -jobs 1 /tmp/linux-3.0
stats.total=36782
0 jakob@brikett:~/code/cshatag$ ./cshatag -recursive -qq -jobs 2 /tmp/linux-3.0
stats.total=36764
6 jakob@brikett:~/code/cshatag$ ./cshatag -recursive -qq -jobs 2 /tmp/linux-3.0
stats.total=36765
6 jakob@brikett:~/code/cshatag$ ./cshatag -recursive -qq -jobs 2 /tmp/linux-3.0
stats.total=36765
6 jakob@brikett:~/code/cshatag$ ./cshatag -recursive -qq -jobs 2 /tmp/linux-3.0
stats.total=36767
6 jakob@brikett:~/code/cshatag$ ./cshatag -recursive -qq -jobs 8 /tmp/linux-3.0
stats.total=36768
6 jakob@brikett:~/code/cshatag$ ./cshatag -recursive -qq -jobs 8 /tmp/linux-3.0
stats.total=36765

[rfjakob]

Oh, you did not change stats.total++ (and similar) in check.go to use atomics.