Support make jobserver protocol

[rossburton]

GNU Make has a neat feature called the jobserver protocol, where the top-level Make can allocate a specific number of job slots, and child makes can take slots to do work in. This was designed to stop the parallelisation problem where a top-level make -j10 may potentially spawn 10 separate sub-makes all with -j10 so there's now 100 parallel jobs.

However, it's also useful for resource control is systems which built multiple pieces of software at once. For example, Bitbake can build N different pieces of software at once, and each of those is passed a -jM flag. If each of these N tasks is compiling then thats's N*M jobs so you don't want N or M to be too high, but if only 1 of N is building then you want M to be high.

With the job server protocol there are N slots in total for all sub makes, so you can control the resource utilisation more accurately. If Samurai also supported the jobserver protocol instead of just -j then it could join in the resource pooling and builds can be more efficient.

The canonical implementation of jobserver is in the GNU Make sources. The documentation is pretty vague as it is considered an implementation detail, but it hasn't changed in 20 years... The prototype BitBake support for a master jobserver queue is at http://git.yoctoproject.org/cgit.cgi/poky-contrib/commit/?h=rpurdie/wipqueue4&id=d66a327fb6189db5de8bc489859235dcba306237

[rossburton]

cc @rpurdie

[bonzini]

Hi, I have done a quick and dirty implementation (less than 200 lines of code) but it uses pthreads. Should I submit a pull request or should I first try to rework it without threads?

[michaelforney]

@bonzini, I am interested in taking a look at what you've got, but I'd definitely prefer an implementation without threads.

[eli-schwartz]

BTW since https://git.savannah.gnu.org/cgit/make.git/commit/?id=7ad2593b2d2bb5b9332f4444d8bf93ac6f958bc6 GNU make supports named pipes as an alternative to anonymous pipes.

Once it actually gets into a release that should be pretty nifty.

[mid-kid]

I feel it's worth noting that gcc picks up on this jobserver when using gcc -flto for LTO builds, and as such is very useful for speeding that up. Using -flto=n to specify the amount of LTO linking jobs isn't great when it comes to projects linking multiple programs concurrently. Oh, and -flto=n still prefers using a jobserver if available.

[bonzini]

@bonzini, I am interested in taking a look at what you've got, but I'd definitely prefer an implementation without threads.

I opened a new PR in #104 now that someone reminded me of it. It does use threads, which is almost mandatory because the GNU Make protocol requires a blocking file descriptor. But the implementation is quite nifty if you ask me. :)

[michaelforney]

It does use threads, which is almost mandatory because the GNU Make protocol requires a blocking file descriptor.

Do you know why the protocol uses blocking file descriptors? That seems like a weird choice to me, and as you say, require the use of threads to implement properly.

[bonzini]

Well it's an old protocol... The idea was that when waiting on a blocking read you had nothing to do anyway; and with some care to avoid races, the read could be interrupted by SIGCHLD (the only other event that Make processed). Of course these days it's a bit inconvenient if you have any kind of event loop; but the solution with the internal pipe, despite the usage of threads, actually simplifies noticeably the handling of the "free" token (which is important to avoid starvation).