Commit graph

16 commits

Author SHA1 Message Date
Avery Pennarun
49ebea445f jwack: don't ever set the jobserver socket to O_NONBLOCK.
It creates a race condition: GNU Make might try to read while the socket is
O_NONBLOCK, get EAGAIN, and die; or else another redo might set it back to
blocking in between our call to make it O_NONBLOCK and our call to read().

This method - setting an alarm() during the read - is hacky, but should work
every time.  Unfortunately you get a 1s delay - rarely - when this happens.
The good news is it only happens when there are no tokens available anyhow,
so it won't affect performance much in any situation I can imagine.
2010-12-10 04:57:13 -08:00
Avery Pennarun
675a5106d2 dup() the jobserver fds to 100,101 to make debugging a bit easier.
Now if a process is stuck waiting on one of those fds, it'll be obvious from
the strace.
2010-12-10 04:11:44 -08:00
Avery Pennarun
6e6e453908 Some speedups for doing redo-ifchange on a large number of static files.
Fix some wastage revealed by the (almost useless, sigh) python profiler.
2010-12-10 00:50:53 -08:00
Avery Pennarun
dcc2edba0c builder.py: further refactoring to run more stuff in the parent process
instead of inside the fork.

Still doesn't seem to affect runtime.  Good.

One nice side effect is jwack.py no longer needs to know anything about our
locks.
2010-11-22 00:04:15 -08:00
Avery Pennarun
7aa7c41e38 builder,jwack: slight cleanup to token passing.
In rare cases, one process could end up holding onto more than one token.
2010-11-21 22:46:20 -08:00
Avery Pennarun
840a8da1ef redo-ifchange: return nonzero if one of the dependencies fails to build.
Oops!  We were just always returning 0 (success) in that case.
2010-11-21 07:15:48 -08:00
Avery Pennarun
116e7e5f13 We now check the lock before releasing our jwack token.
That way, if everything is locked, we can determine that with a single
token, reducing context switches.

But mostly this is good because the code is simpler.
2010-11-19 07:20:55 -08:00
Avery Pennarun
362ca2997a A whole bunch of cleanups to state.Lock.
Now t/curse passes again when parallelized (except for the countall
mismatch, since we haven't fixed the source of that problem yet).  At least
it's consistent now.

There's a bunch of stuff rearranged in here, but the actual important
problem was that we were doing unlink() on the lock fifo even if ENXIO,
which meant a reader could connect in between ENXIO and unlink(), and thus
never get notified of the disconnection.  This would cause the build to
randomly freeze.
2010-11-19 06:07:41 -08:00
Avery Pennarun
2a9a332451 jwack.py: print the full traceback if a task fails to run. 2010-11-19 00:54:36 -08:00
Avery Pennarun
94b0e7166e Move atoi() into atoi.py and add a new debug2() debug level.
atoi() was getting redundant, and unfortunately we can't easily load
helpers.py in some places where we'd want to, because it depends on vars.py.
So move it to its own module.
2010-11-16 04:13:17 -08:00
Avery Pennarun
93ab32d0f9 jwack: has not been actually executable since the earlier rewrite. 2010-11-16 00:28:01 -08:00
Avery Pennarun
3803da525c Notice sooner when make has "helpfully" closed its job control file descriptors. 2010-11-13 05:05:48 -08:00
Avery Pennarun
662f53896a _try_read: hacky fix to make GNU make less angry.
We'll have to stop using nonblocking reads, unfortunately.  But this seems
to work better than nothing.  There's still a race condition that could
theoretically make GNU make angry, unfortunately, since we briefly set the
socket to nonblocking.
2010-11-13 04:50:53 -08:00
Avery Pennarun
bd5daf9754 Totally disgusting support for jobservers.
It needs some heavy rethinking and cleanup.  But it seems to work!  And it's
even compatible with GNU make, apparently!
2010-11-13 04:36:44 -08:00
Avery Pennarun
7cbf39d52a Make jwack... mostly... work with GNU make.
But it seems to be pretty unsolvable in the current form; the problem is
that when you're nesting one jwack inside the other and the jobserver is GNU
make, there's no way to tell the parent jwack not to use up a token.  Thus,
if you nest too deeply, it just deadlocks.

So this approach isn't really going to work the way it is.
2010-11-12 21:09:29 -08:00
Avery Pennarun
f77e4b5c91 Add jwack, a GNU make-like jobserver.
Theoretically compatible with GNU make's jobserver pipes.  Haven't tested
that yet.
2010-11-12 20:10:21 -08:00