The way the code was written, we'd give up our token, detect a cyclic
dependency, and then try to get our token back before exiting. Even
with -j1, the temporary token release allowed any parent up the tree to
continue running jobs, so it would take an arbitrary amount of time
before we could exit (and report an error code to the parent).
There was no visible symptom of this except that, with -j1, t/355-deps-cyclic
would not finish until some of the later tests finished, which was
surprising.
To fix it, let's just check for a cyclic dependency first, then release
the token only once we're sure things are sane.
This happens sometimes, for example, if you do
whatever | while read x; do
redo-ifchange "$x"
done
and the input contains blank lines.
We could ignore the request for blankness, but it seems like that
situation might indicate a more serious bug in your parser, so it's
probably better to just abort with a meaningful error.
If we tried to build target a/b/c/d and a/b/c didn't exist yet, we
would correctly name the temp file something like a__b/c__d.tmp. But
if a/b didn't exist yet, we named the temp file a/b__c/d.tmp, which
didn't work. Instead, name it a/b__c__d.tmp as expected.
With the new "continue" feature on by default, it turned out that
ctrl-c during a build, or a .do file returning an error, would mark a
target as "built" even though it hadn't been. This would prevent
retrying it when you started minimal/do again. Use a temp file
instead.
It's a little tricky: to prevent accidental recursion, we want to
create a file *before* building, but clean up that file when starting
the next session. And we rename that file to the actual .did file
*after* building successfully.
If redo-ifchange is a symlink to minimal/do, and we run it from
*outside* an existing session, it shouldn't default to 'all'.
redo-sources, redo-targets, and redo-ood aren't too useful with
minimal/do, but they should exist in case someone tries to use them, so
just symlink them to /bin/true, like we do with redo-ifcreate.
It would be incorrect to print ../$1.do (we only use $1.do in the
*current* directly, not prefix dirs; we only use default*.do in
those). However, when we found ../default.do, we forgot to print it,
because of a silly logic error.
Adding redo-whichdo wasn't so hard, except that it exposed a bunch of
mistakes in the way minimal/do was choosing .do files. Most of the
wrong names it tried didn't matter (since they're very unlikely to
exist) except that they produce the wrong redo-whichdo output.
Debugging that took so much effort that I added unit tests for some
functions to make it easier to find problems in the future.
Note: this change apparently makes minimal/do about 50% slower than
before, because the _find_dofiles algorithm got less optimized (but
more readable). It's still not very slow, and anyway, since it's
minimal/do, I figured readable/correct code probably outweighed a bit
of speed. (Although if anyone can come up with an algorithm that's
clear *and* works better, I won't turn it down :) I feel like I must
be doing it the hard way...)
After playing with "continuable" mode for a while, it seems to me that
what most people want will be incremental builds by default (albeit
without any dependency checking to rebuild a target if sources change).
This lets you run minimal/do several times in a row to incrementally
build up your project while you're playing around. If you want to
start over, use -c.
As a result, in test.do we now recommend using 'minimal/do -c test'
since minimal/do now strays even further from proper redo semantics
otherwise.
While we're here, update the usage message to describe what all the new
options mean.
When we can't find a .do file, we walk all the way back to the root
directory. When that happens, the root directory is actually searched
twice. This is harmless (since a .do file doesn't exist there anyway)
but causes redo-whichdo to produce the wrong output.
Also, add a test, which I forgot to do when writing whichdo in the
first place.
To make the test work from the root directory, we need a way to
initialize redo without actually creating a .redo directory. Add a
init_no_state() function for that purpose, and split the necessary path
functions into their own module so we can avoid importing builder.py.
The new format is just a list of .do files we tried, with a newline
after each one. If we successfully found a .do file, we exit 0, else
we exit 1.
As discussed on the redo-list mailing list, it's easier to parse
without the extra cruft. This makes users figure out $1 and $2
themselves, but that's not very hard, and maybe for the best.
We already printed an error at build time, but added the broken
dependency anyway. If the .do script decided to succeed despite
redo-ifchange aborting, the target would be successfully created
and we'd end up with an infinite loop when running isdirty() later.
The result was still "correct", because python helpfully aborted
the infinite loop after the recursion got too deep. But let's
explicitly detect it and print a better error message.
(Thanks to Nils Dagsson Moskopp's redo-testcases repo for exposing this
problem. If you put a #!/bin/sh header on your .do script means you
need to run 'set -e' yourself if you want .do scripts to abort after an
error, which you almost always do, and those testcases don't, which
exposed this bug if you ran the tests twice.)
This is the test that
x=y f
does *not* unset x after running f, if f is a shell function. Apparently
that's the right thing to do, but freebsd dash 0.5.10.2 fails it. This is
surprising because debian dash 0.5.8-2.4 passes, and it is presumably older.
Downgrade this to a warning because we want freebsd to have some cheap sh
variant that passes with redo, and nobody should be *relying* on this
insane behaviour anyway.
freebsd 11.2 sh (but not freebsd dash) fails test #24. That one seems
rather serious, so I don't want to downgrade it to a warning.
If we end up in builder phase 2, where we might need to
build stuff that was previously locked by someone else,
we will need to obtain a job token *and* the lock at the
same time in order to continue. To prevent deadlocks,
we don't wait synchronously for one lock while holding the
other.
If several instances are fighting over the same lock and
there are insufficient job tokens for everyone, timing
could cause them to fight for a long time. This seems
to happen a lot in freebsd for some reason. To be a good
citizen, sleep for a while after each loop iteration.
This should ensure that eventually, most of the fighting
instances will be asleep by the time the next one tries to
grab the token, thus breaking the deadlock.
-x, -v, and -d are the same as redo.
-c means "continuable", which disables the feature that deletes (and
forgets) all targets at the start of each run. This is a little risky,
since minimal/do still doesn't understand dependencies, but it allows
you to run minimal/do several times in succession, so that
minimal/do -c a
minimal/do -c b
is the same as
minimal/do a b
We previously assumed that redo and redo-ifchange are the same in
minimal/do's design, because it rebuilds all targets on every run, and
so there's no reason to ever build the same target more than once.
Unfortunately that's incorrect: if you run 'redo x' from two points in
a single run (or even twice in the same .do file), we expect x to be
built twice. If you wanted redo to decide whether to build it the
second time, you should have used redo-ifchange.
t/102-empty/touchtest was trying to test for this. However, a
second bug in minimal/do made the test pass anyway. minimal/do would
*always* rebuild any target x that produced no output, not caring
whether it had tried to build before, whether you used redo or
redo-ifchange. And while we tested that redo would redo a file that
had been deleted, we didn't ensure that it would redo a file that was
*not* deleted, nor that redo-ifchange would *not* redo that file.
Fix both bugs in minimal/do, and make t/102-empty/touchtest cover the
missing cases.
We need to create the File object to get its f.id, then lock that id.
During that gap, another instance of redo may have modified the file or
its state data, so we have to refresh it.
This fixes 'redo -j10 t/stress'.
It looks like we're updating the stamp for t/countall while another
task is replacing the file, which suggests a race condition in our
state management database.
The first time we notice a file has been overridden, log the old and
new stamp data, which might give a hint about how this happened.
Currently if I do
rm t/950-curse/countall
while :; do redo -j10 t/950-curse/all --shuffle || break; done
it will end up complaining that countall has been overridden within
just a few runs, even though it definitely hasn't been. There seems to
be someone reading a file stamp while someone else is redoing the
file, but I haven't found it yet.
flush-cache can cause files affected by redo-stamp to get rebuilt
unnecessarily, which the test is specifically trying to validate.
Since other tests run flush-cache at random times when using -j, this
would cause random test failures.
Because the two programs use separate state databases, it helps if we
clean up some temp files between runs. Otherwise they might think you
created some targets "by hand" and refuse to rebuild them.
I'm not quite sure what I was thinking, using redo-ifchange there, but
the result was that some tests wouldn't run if you run 'redo test'
repeatedly, even after modifying redo itself.
Also tweaked t/950-curse so that it always runs, not just the first
time.
We grew a test for these at some point, but minimal/do didn't actually
pass it.
sh syntax oddity: if you say
x=$1 y=$2
then it works fine when $1 and $2 contain spaces. But if you say
export x=$1 y=$2
(or "local" instead of "export") then $1 and $2 will be split on IFS,
and it won't do what you think. I guess this is because 'export' and
'local' are like commands, and command arguments are split if not
quoted.
I feel a little dirty doing this, but the way the code was before, redo
almost always picked bash as the shell. bash is way too overpowered
and this led to bashisms in do scripts unnecessarily. The two failures
in dash are things that I would really like to have, but they haven't
materialized after 6 years, so I guess we should be realistic.
To appropriately penalize bash for asking for trouble, I added a
warning about [ 1 == 1 ] syntax being valid (as opposed to the POSIX
correct [ 1 = 1 ]). This allows dash to be selected ahead of bash.
I also moved 'sh' to the end of the list, because although it's the
weakest shell on some systems, on other systems it's just bash. And I
put zsh in front of bash, because fewer people have zsh and we want
them to test zsh.
The builder was holding lock variables in the loop which means that
sometimes a state.Lock object would be created for the same file-id
twice, triggering the assertion. Assign the lock variables to None to
ensure that the state.Lock objects are destroyed before creating the
next one in the loop.
If a depends on b which depends on a, redo would just freeze. Now it
aborts with a somewhat helpful error message.
[Updated by apenwarr for coding style and to add a test.]
Removed information (ctime) that was causing spurious
"you modified it" warnings.
[apenwarr: This was not quite right. ctime includes some things we
don't want to care about, such as link count, so we have to remove it.
But it also notices changes to st_uid, st_gid, and st_mode, which we do
care about, so now we have to include those explicitly.]
[apenwarr's note: ctime includes extra inode attributes like link
count, which are not important for this check, but which could cause
spurious warnings.]
The >& form is only for file descriptors, passing a file name there is
a bash extension.
$ /bin/dash -c 'echo foo >&/dev/null'
/bin/dash: 1: Syntax error: Bad fd number
WAL mode does make the deadlocking on MacOS go away. This suggests
that pysqlite3 was leaving *read* transactions open for a long time; in
old-style sqlite journals, this prevents anyone from obtaining a write
lock, although it doesn't prevent other concurrent reads.
With WAL journals, writes can happen even while readers are holding a
lock, but the journal doesn't flush until the readers have released it.
This is not a "real" fix but it's fairly harmless, since all redo
instances will exit eventually, and when they do, the WAL journal can
be flushed.
I think we were sometimes leaving half-done sqlite transactions sitting
around for a long time (eg. across sub-calls to .do files). This
seemed to be okay on Linux, but caused sqlite deadlocks on MacOS. Most
likely it's not the operating system, but the sqlite version and
journal mode in use.
In any case, the correct thing to do is to actually commit or rollback
transactions, not leave them hanging around.
...unfortunately this doesn't actually fix my MacOS deadlocks, which
makes me rather nervous.
If ./default.do knows how to build x/y/z, then we will run
./default.do x/y/z x/y/z x__y__z.redo2.tmp
which can correctly generate $3, but then we can fail to rename it to
x/y/z because x/y doesn't exist. This would previously through an
exception. Now it prints a helpful error message.
default.do may create x/y, in which case renaming will succeed.
