This is slightly inelegant, as the old style
echo foo
echo blah
chmod a+x $3
doesn't work anymore; the stuff you wrote to stdout didn't end up in $3.
You can rewrite it as:
exec >$3
echo foo
echo blah
chmod a+x $3
Anyway, it's better this way, because now we can tell the difference between
a zero-length $3 and a nonexistent one. A .do script can thus produce
either one and we'll either delete the target or move the empty $3 to
replace it, whichever is right.
As a bonus, this simplifies our detection of whether you did something weird
with overlapping changes to stdout and $3.
Although we were deadlock-free before, under some circumstances we'd end up
holding a perfectly good token while in sync wait; that would reduce our
parallelism for no good reason. So give back our tokens before waiting for
anybody else.
That way the user can modify an auto-generated 'compile' script, for
example, and it'll stay modified.
If they delete the file, we can then generate it for them again.
Also, we have to warn whenever we're doing this, or people might think it's
a bug.
It's really a separate condition. And since we're not removing the target
*file* in case of error - we update it atomically, and keeping it is better
than losing it - there's no reason to wipe the timestamp in that case
either.
However, we do need to know that the build failed, so that anybody else
(especially in a parallel build) who looks at that target knows that it
died. So add a separate flag just for that.
This should reduce filesystem grinding a bit, and makes the code simpler.
It's also theoretically a bit more portable, since I'm guessing fifo
semantics aren't the same on win32 if we ever get there.
Also, a major problem with the old fifo-based system is that if a redo
process died without cleaning up after itself, it wouldn't delete its
lockfiles, so we had to wipe them all at the beginning of each build. Now
we don't; in theory, you can now have multiple copies of redo poking at the
same tree at the same time and not stepping on each other.
Just commit when we're about to do something blocking. sqlite goes a lot
faster with bigger transactions. This change does show a small percentage
speedup in tests, but not as much as I'd like.
It passes all tests when run serialized, but still gives weird errors
(OperationalError: database is locked) when run with -j5. sqlite3 shouldn't
be barfing just because the database is locked, since the default timeout is
5 seconds, and it's dying *way* faster than that.
This allows files to transition from generated to not-generated if the .do
file is ever removed (ie. the user is changing things and the file is now a
source file, not a target).
So if you have a default.do, it might be used to build mydir, even if mydir
already existed when we started.
This might be useful if you have a mydir.setup.do and a mydir.do; mydir.do
depends on mydir.setup.do, but mydir.setup.do creates mydir, it just doesn't
*finish* with mydir. Still, my the time mydir.do runs, mydir already
exists, and redo would get confused.
I think directories are fundamentally special in this way, because it makes
sense to "create" a directory even if that directory isn't "done" at this
phase.
The interaction of REDO_STARTDIR, REDO_PWD, and getcwd() are pretty
complicated. In this case, we accidentally assumed that the current
instance of redo was running with getcwd() == REDO_STARTDIR+REDO_PWD, and so
the new target was REDO_STARTDIR+REDO_PWD+t, but this isn't the case if the
current .do script did chdir().
The correct answer is REDO_STARTDIR+getcwd()+t.
If a and b both depend on c, and c is a static (non-generated) file that has
changed since the last successful build of a and b, we would try to redo
a, but would forget to redo b. Now it does both.
If a file previously was generated but now isn't (ie. its .do file
disappears), we would never re-stamp that target, and so all its
dependencies would rebuild continually.
It actually decreases readability of the .do files - by not making it
explicit when you're going into a subdir.
Plus it adds ambiguity: what if there's a dirname.do *and* a dirname/all?
We could resolve the ambiguity if we wanted, but that adds more code, while
taking out this special case makes *less* code and improves readability.
I think it's the right way to go.
Normally, creating the target $1 yourself is bad; create $3 instead. But if
$1 is a directory, we'll allow it. That way 'redo subdir' can call
subdir.do, and subdir.do can both create the directory *and* run a bunch of
sub-.do files on it.
...because we deliberately stamp non-generated files as well, and that
doesn't need to imply that we rebuilt them just now. In fact, we know for a
fact that we *didn't* rebuild them just now, but we still need to record the
timestamp for later.
.do files should never modify $1, and should write to *either* $3 or stdout,
but not both. If they write to both, it's probably because they forgot to
redirect stdout to stderr, a very easy mistake to make but a hard one to
detect.
Now redo detects it for you and prints an informative message.
If 'redo clean' deletes the lockfile after trylock() succeeds but before
unlock(), then unlock() won't be able to open the pipe in order to release
readers, and any waiters might end up waiting forever.
We can't open the fifo for write until there's at least one reader, so let's
open a reader *just* to let us open a writer. Then we'll leave them open
until the later unlock(), which can just close them both.
Otherwise it could be inherited by other jwack jobs started from the same
parent process, resulting in some very-hard-to-debug race conditions, let me
tell you.
This avoids a second fork altogether, which apparently doesn't matter all
that much (5.3s -> 5.15s). However, simply not importing the subprocess
module reduces us further to 4.6s.
And 'ps axf' now looks prettier because we have fewer stupid intermediate
processes.
instead of inside the fork.
Still doesn't seem to affect runtime. Good.
One nice side effect is jwack.py no longer needs to know anything about our
locks.
We can also avoid forking altogether if should_build() returns false. This
doesn't seem to result in any noticeable speedup, but it's cleaner at least.
Previously, the default was to *always* keep going, which is actually not
usually what you want. Now we actually exit correctly after an error. Of
course you still might have multiple errors before existing if you were
building in parallel.
Get rid of the "locked..." and "...unlocked!" messages by default, since
they're not usually interesting. But add a new option to bring them back in
case we end up with trouble debugging the locking stuff. (I don't really
100% trust it yet, although I haven't had a problem for a while now.)
