Parallelism and redo-log cause lots of confusion for any rules that try
to ask the user for questions, so disable it altogether.
Arguably, we should just disable stdin all the time, but maybe it's
still occasionally useful (even though you have to pass --no-log to get
it back).
I attempted to fix this in commit c06d1fba40, but it was apparently
incomplete and not all cases were covered by tests.
Let's add a much more thorough test by going through every possible
combination and making sure redo-{sources,targets,ood} all work as
expected, that the "you modified it" warning does or does not show up
when expected, and that dependencies are rebuilt the number of times we
expect.
It seems like we're using these differently than most readthedocs.org
users. Remove the borders and padding so they work better inline, and
prevent confusing word wraps.
This shows how to dynamically generate a plot in R+ggplot2, then embed
it into a latex document, and compile it to pdf, all with proper
autodependencies.
Mailing list discussion was here:
https://groups.google.com/forum/#!topic/redo-list/wLMZMxtn4wo
Several more contributors replied to me personally to say that they
don't have a problem with the change (and several consider the change
to be an improvement).
The overwhelming majority of everything in the redo repo was written by
me, so I have the right to change the license unilaterally anyway,
subject to a few rules. Since the new license actually removes
licensing/usage restrictions for everyone, this should be no problem.
This was required for old versions of MacOS X, but the bug has now been
fixed, so let's de-clutter the .redo directory by reducing to a single
lock file instead of one per fid.
This reverts commit 08b05fd72f92fa0061b3d082b391151b43cd7497.
When we check dependencies and a previously-is_generated dependency
existed before, but no longer does, forget that it was is_generated.
This slightly improves the situation where as a project evolves, a file
that used to be a target gets removed, and then later is re-added as a
static source file. (It doesn't fix the other variant, where a file is
changed from target to source in a single atomic change, and is never
missing. That one will be trickier to handle.)
While adding a test for this behaviour, I discovered that redo-sources,
redo-targets, and redo-ood were reporting their output relative to
STARTDIR instead of relative to $PWD, so fix that too.
flush-cache reduces the failed_runid by 1 each time, and it runs
multiple times per 'redo test'. if failed_runid goes to zero, it would
be treated as success ("no failure") rather than a real failure at
runid 0.
Extremely rarely, if that target had not yet previously been added to
the redo database, redo-log would manage to run before the database got
flushed, and then it would complain that the file is "not known to
redo" and abort.
I had to run "git clean -fdx; redo -j11 test" in a loop several hundred
times before this race condition triggered, but it did trigger eventually.
This avoids a mkdocs bug where mkdocs will (rarely) see redo temp files
in its directory list, but they're gone by the time it tries to open
them, so it aborts with an exception.
In commit redo-0.11-4-g34669fb, we changed os.stat into os.lstat to
avoid false positives in the "manual override" detector: a .do file
that generates $3 as a symlink would trigger manual override if the
*target* of that symlink ever changed, which is incorrect.
Unfortunately using os.lstat() leads to a different problem: if X
depends on Y and Y is a symlink to Z, then X would not be rebuilt when
Z changes, which is clearly wrong.
The fix is twofold:
1. read_stamp() should change on changes to both the link itself,
*and* the target of the link.
2. We shouldn't mark a target as overridden under so many situations.
We'll use *only* the primary mtime of the os.lstat(), not all the
other bits in the stamp.
Step 2 fixes a few other false positives also. For example, if you
'cp -a' a whole tree to another location, the st_ino of all the targets
will change, which would trigger a mass of "manual override" warnings.
Although a change in inode is sufficient to count an input as having
changed (just to be extra safe), it should *not* be considered a manual
override. Now we can distinguish between the two.
Because the stamp format has changed, update the SCHEMA_VER field. I
should have done this every other time I changed the stamp format, but
I forgot. Sorry. That leads to spurious "manually modified" warnings
after upgrading redo.
* redo-log:
redo-log: add man page.
redo-log: add automated tests, and fix some path bugs revealed by them.
redo-log: fix stdout vs stderr; don't recapture if .do script redirects stderr.
redo-log: don't show status line until >1.0 seconds after starting.
Add --color and --no-color options.
redo-log: --debug-pids works properly again.
Split --raw-logs into --no-pretty and --no-log options.
redo-log: prioritize the "foreground" process.
redo-log: correctly indent first level of recursion.
Raw logs contain @@REDO lines instead of formatted data.
redo-log: status line should use actual terminal width.
redo-log: capture and linearize the output of redo builds.
Use signal.setitimer instead of signal.alarm.
When a log for X was saying it wanted to refer to Y, we used a relative
path, but it was sometimes relative to the wrong starting location, so
redo-log couldn't find it later.
Two examples:
- if default.o.do is handling builds for a/b/x.o, and default.o.do
does 'redo a/b/x.h', the log for x.o should refer to ./x.h, not
a/b/x.h.
- if foo.do is handling builds for foo, and it does
"cd a/b && redo x", the log for foo should refer to a/b/x, not just
x.
redo-log should log to stdout, because when you ask for the specific
logs from a run, the logs are the output you requested. redo-log's
stderr should be about any errors retrieving that output.
On the other hand, when you run redo, the logs are literally the stderr
of the build steps, which are incidental to the main job (building
things). So that should be send to stderr. Previously, we were
sending to stderr when --no-log, but stdout when --log, which is
totally wrong.
Also, adding redo-log had the unexpected result that if a .do script
redirected the stderr of a sub-redo or redo-ifchange to a file or pipe,
the output would be eaten by redo-log instead of the intended output.
So a test runner like this:
self.test:
redo self.runtest 2>&1 | grep ERROR
would not work; the self.runtest output would be sent to redo's log
buffer (and from there, probably printed to the toplevel redo's stderr)
rather than passed along to grep.
--no-log: don't capture logs or run redo-log (same as pre-redo-log redo)
--no-pretty: don't pretty-print logs, print @@REDO lines.
The latter is an option to both redo and redo-log.
When running a parallel build, redo-log -f (which is auto-started by
redo) tries to traverse through the logs depth first, in the order
parent processes started subprocesses. This works pretty well, but if
its dependencies are locked, a process might have to give up its
jobserver token while other stuff builds its dependencies. After the
dependency finishes, the parent might not be able to get a token for
quite some time, and the logs will appear to stop.
To prevent this from happening, we can instantiate up to one "cheater"
token, only in the foreground process (the one locked by redo-log -f),
which will allow it to continue running, albeit a bit slowly (since it
only has one token out of possibly many). When the process finishes,
we then destroy the fake token. It gets a little complicated; see
explanation at the top of jwack.py.
This makes them more reliable to parse. redo-log can parse each line,
format and print it, then recurse if necessary. This got a little ugly
because I wanted 'redo --raw-logs' to work, which we want to format the
output nicely, but not call redo-log.
(As a result, --raw-logs has a different meaning to redo and
redo-log, which is kinda dumb. I should fix that.)
As an added bonus, redo-log now handles indenting of recursive logs, so
if the build was a -> a/b -> a/b/c, and you look at the log for a/b, it
can still start at the top level indentation.
redo now saves the stderr from every .do script, for every target, into
a file in the .redo directory. That means you can look up the logs
from the most recent build of any target using the new redo-log
command, for example:
redo-log -r all
The default is to show logs non-recursively, that is, it'll show when a
target does redo-ifchange on another target, but it won't recurse into
the logs for the latter target. With -r (recursive), it does. With -u
(unchanged), it does even if redo-ifchange discovered that the target
was already up-to-date; in that case, it prints the logs of the *most
recent* time the target was generated.
With --no-details, redo-log will show only the 'redo' lines, not the
other log messages. For very noisy build systems (like recursing into
a 'make' instance) this can be helpful to get an overview of what
happened, without all the cruft.
You can use the -f (follow) option like tail -f, to follow a build
that's currently in progress until it finishes. redo itself spins up a
copy of redo-log -r -f while it runs, so you can see what's going on.
Still broken in this version:
- No man page or new tests yet.
- ANSI colors don't yet work (unless you use --raw-logs, which gives
the old-style behaviour).
- You can't redirect the output of a sub-redo to a file or a
pipe right now, because redo-log is eating it.
- The regex for matching 'redo' lines in the log is very gross.
Instead, we should put the raw log files in a more machine-parseable
format, and redo-log should turn that into human-readable format.
- redo-log tries to "linearize" the logs, which makes them
comprehensible even for a large parallel build. It recursively shows
log messages for each target in depth-first tree order (by tracing
into a new target every time it sees a 'redo' line). This works
really well, but in some specific cases, the "topmost" redo instance
can get stuck waiting for a jwack token, which makes it look like the
whole build has stalled, when really redo-log is just waiting a long
time for a particular subprocess to be able to continue. We'll need to
add a specific workaround for that.
I also cleaned up the installation section and added links to various
competing redo implementations.
The new README.md is basically just link to the docs on
readthedocs.org, and a link to the mailing list.
These docs need a *lot* more work, but this is enough of an improvement
that I'll commit it anyway for now.
Replaced all instances of 'python' with 'python2'
I think it's quite dumb that arch linux *ever* changed the default /usr/bin/python to python3, but oh well. This patch seems to be okay now that even debian provides a /usr/bin/python2 symlink for the last several years.
No content yet other than man pages. To make the man pages render
nicely in mkdocs, I removed the static %-headers and put the code for
them into default.md.tmp.do instead. That way, the raw input redo-*.md
files will look right in mkdocs.
On systems where 'python' refers to python3, redo
failed to launch. All invocations of python have been
made explicitly python2 invocations. All tests pass
on an Arch Linux system as of this commit.
The way the code was written, we'd give up our token, detect a cyclic
dependency, and then try to get our token back before exiting. Even
with -j1, the temporary token release allowed any parent up the tree to
continue running jobs, so it would take an arbitrary amount of time
before we could exit (and report an error code to the parent).
There was no visible symptom of this except that, with -j1, t/355-deps-cyclic
would not finish until some of the later tests finished, which was
surprising.
To fix it, let's just check for a cyclic dependency first, then release
the token only once we're sure things are sane.
This happens sometimes, for example, if you do
whatever | while read x; do
redo-ifchange "$x"
done
and the input contains blank lines.
We could ignore the request for blankness, but it seems like that
situation might indicate a more serious bug in your parser, so it's
probably better to just abort with a meaningful error.
If we tried to build target a/b/c/d and a/b/c didn't exist yet, we
would correctly name the temp file something like a__b/c__d.tmp. But
if a/b didn't exist yet, we named the temp file a/b__c/d.tmp, which
didn't work. Instead, name it a/b__c__d.tmp as expected.
With the new "continue" feature on by default, it turned out that
ctrl-c during a build, or a .do file returning an error, would mark a
target as "built" even though it hadn't been. This would prevent
retrying it when you started minimal/do again. Use a temp file
instead.
It's a little tricky: to prevent accidental recursion, we want to
create a file *before* building, but clean up that file when starting
the next session. And we rename that file to the actual .did file
*after* building successfully.
