Now that the python scripts are all in a "redo" python module, we can
use the "new style" (ahem) package-relative imports. This appeases
pylint, plus avoids confusion in case more than one package has
similarly-named modules.
This removes another instance of magical code running at module import
time. And the process title wasn't really part of the state database
anyway.
Unfortunately this uncovered a bug: the recent change to use
'python -S' makes it not find the setproctitle module if installed.
My goodness, I hate the horrible python easy_install module gunk that
makes startup linearly slower the more modules you have installed,
whether you import them or not, if you don't use -S. But oh well,
we're stuck with it for now.
Merge the two files into env, and make each command explicitly call the
function that sets it up in the way that's needed for that command.
This means we can finally just import all the modules at the top of
each file, without worrying about import order. Phew.
While we're here, remove the weird auto-appending-'all'-to-targets
feature in env.init(). Instead, do it explicitly, and only from redo and
redo-ifchange, only if is_toplevel and no other targets are given.
Previously, we'd try to build all the critical stuff first, and then
run the tests. Nowadays, it takes a little longer to build the docs
(especially some of the docs/cookbook/ stuff), and this isn't needed to
run the tests, so let's allow them to parallelize.
This avoids a name overlap with the system-installed copy of python.
Since redo adds the redo/ dir to the $PATH before running .do files,
python.do might see its own previously-created target instead of the
"real" python when testing, and create an infinite loop by accident.
They really aren't locks at all, they're a cycle detector. Also rename
REDO_LOCKS to a more meaningful REDO_CYCLES. And we'll move the
CyclicDependencyError exception in here as well, instead of state.py
where it doesn't really belong.
It's time to start preparing for a version of redo that doesn't work
unless we build it first (because it will rely on C modules, and
eventually be rewritten in C altogether).
To get rolling, remove the old-style symlinks to the main programs, and
rename those programs from redo-*.py to redo/cmd_*.py. We'll also move
all library functions into the redo/ dir, which is a more python-style
naming convention.
Previously, install.do was generating wrappers for installing in
/usr/bin, which extend sys.path and then import+run the right file.
This made "installed" redo work quite differently from running redo
inside its source tree. Instead, let's always generate the wrappers in
bin/, and not make anything executable except those wrappers.
Since we're generating wrappers anyway, let's actually auto-detect the
right version of python for the running system; distros can't seem to
agree on what to call their python2 binaries (sigh). We'll fill in the
right #! shebang lines. Since we're doing that, we can stop using
/usr/bin/env, which will a) make things slightly faster, and b) let us
use "python -S", which tells python not to load a bunch of extra crap
we're not using, thus improving startup times.
Annoyingly, we now have to build redo using minimal/do, then run the
tests using bin/redo. To make this less annoying, we add a toplevel
./do script that knows the right steps, and a Makefile (whee!) for
people who are used to typing 'make' and 'make test' and 'make clean'.
Because our experiment involves creating a file called redo/sh, and the
redo directory (or some other version of redo's dirctory) might be on
the path, just testing 'sh' is actually wrong: we might end up with
some earlier version of redo-sh.
Instead, specifically for sh, always demand that we test /bin/sh.
This seems to only affect old zsh on MacOS. But we want to catch it
anyway, because it caused t/351-deps-forget to fail in a weird way on
that version of zsh.
Shells really suck.
Parallelism and redo-log cause lots of confusion for any rules that try
to ask the user for questions, so disable it altogether.
Arguably, we should just disable stdin all the time, but maybe it's
still occasionally useful (even though you have to pass --no-log to get
it back).
I attempted to fix this in commit c06d1fba40, but it was apparently
incomplete and not all cases were covered by tests.
Let's add a much more thorough test by going through every possible
combination and making sure redo-{sources,targets,ood} all work as
expected, that the "you modified it" warning does or does not show up
when expected, and that dependencies are rebuilt the number of times we
expect.
It seems like we're using these differently than most readthedocs.org
users. Remove the borders and padding so they work better inline, and
prevent confusing word wraps.
This shows how to dynamically generate a plot in R+ggplot2, then embed
it into a latex document, and compile it to pdf, all with proper
autodependencies.
Mailing list discussion was here:
https://groups.google.com/forum/#!topic/redo-list/wLMZMxtn4wo
Several more contributors replied to me personally to say that they
don't have a problem with the change (and several consider the change
to be an improvement).
The overwhelming majority of everything in the redo repo was written by
me, so I have the right to change the license unilaterally anyway,
subject to a few rules. Since the new license actually removes
licensing/usage restrictions for everyone, this should be no problem.
This was required for old versions of MacOS X, but the bug has now been
fixed, so let's de-clutter the .redo directory by reducing to a single
lock file instead of one per fid.
This reverts commit 08b05fd72f92fa0061b3d082b391151b43cd7497.
When we check dependencies and a previously-is_generated dependency
existed before, but no longer does, forget that it was is_generated.
This slightly improves the situation where as a project evolves, a file
that used to be a target gets removed, and then later is re-added as a
static source file. (It doesn't fix the other variant, where a file is
changed from target to source in a single atomic change, and is never
missing. That one will be trickier to handle.)
While adding a test for this behaviour, I discovered that redo-sources,
redo-targets, and redo-ood were reporting their output relative to
STARTDIR instead of relative to $PWD, so fix that too.
flush-cache reduces the failed_runid by 1 each time, and it runs
multiple times per 'redo test'. if failed_runid goes to zero, it would
be treated as success ("no failure") rather than a real failure at
runid 0.
Extremely rarely, if that target had not yet previously been added to
the redo database, redo-log would manage to run before the database got
flushed, and then it would complain that the file is "not known to
redo" and abort.
I had to run "git clean -fdx; redo -j11 test" in a loop several hundred
times before this race condition triggered, but it did trigger eventually.
This avoids a mkdocs bug where mkdocs will (rarely) see redo temp files
in its directory list, but they're gone by the time it tries to open
them, so it aborts with an exception.
In commit redo-0.11-4-g34669fb, we changed os.stat into os.lstat to
avoid false positives in the "manual override" detector: a .do file
that generates $3 as a symlink would trigger manual override if the
*target* of that symlink ever changed, which is incorrect.
Unfortunately using os.lstat() leads to a different problem: if X
depends on Y and Y is a symlink to Z, then X would not be rebuilt when
Z changes, which is clearly wrong.
The fix is twofold:
1. read_stamp() should change on changes to both the link itself,
*and* the target of the link.
2. We shouldn't mark a target as overridden under so many situations.
We'll use *only* the primary mtime of the os.lstat(), not all the
other bits in the stamp.
Step 2 fixes a few other false positives also. For example, if you
'cp -a' a whole tree to another location, the st_ino of all the targets
will change, which would trigger a mass of "manual override" warnings.
Although a change in inode is sufficient to count an input as having
changed (just to be extra safe), it should *not* be considered a manual
override. Now we can distinguish between the two.
Because the stamp format has changed, update the SCHEMA_VER field. I
should have done this every other time I changed the stamp format, but
I forgot. Sorry. That leads to spurious "manually modified" warnings
after upgrading redo.
* redo-log:
redo-log: add man page.
redo-log: add automated tests, and fix some path bugs revealed by them.
redo-log: fix stdout vs stderr; don't recapture if .do script redirects stderr.
redo-log: don't show status line until >1.0 seconds after starting.
Add --color and --no-color options.
redo-log: --debug-pids works properly again.
Split --raw-logs into --no-pretty and --no-log options.
redo-log: prioritize the "foreground" process.
redo-log: correctly indent first level of recursion.
Raw logs contain @@REDO lines instead of formatted data.
redo-log: status line should use actual terminal width.
redo-log: capture and linearize the output of redo builds.
Use signal.setitimer instead of signal.alarm.
When a log for X was saying it wanted to refer to Y, we used a relative
path, but it was sometimes relative to the wrong starting location, so
redo-log couldn't find it later.
Two examples:
- if default.o.do is handling builds for a/b/x.o, and default.o.do
does 'redo a/b/x.h', the log for x.o should refer to ./x.h, not
a/b/x.h.
- if foo.do is handling builds for foo, and it does
"cd a/b && redo x", the log for foo should refer to a/b/x, not just
x.
redo-log should log to stdout, because when you ask for the specific
logs from a run, the logs are the output you requested. redo-log's
stderr should be about any errors retrieving that output.
On the other hand, when you run redo, the logs are literally the stderr
of the build steps, which are incidental to the main job (building
things). So that should be send to stderr. Previously, we were
sending to stderr when --no-log, but stdout when --log, which is
totally wrong.
Also, adding redo-log had the unexpected result that if a .do script
redirected the stderr of a sub-redo or redo-ifchange to a file or pipe,
the output would be eaten by redo-log instead of the intended output.
So a test runner like this:
self.test:
redo self.runtest 2>&1 | grep ERROR
would not work; the self.runtest output would be sent to redo's log
buffer (and from there, probably printed to the toplevel redo's stderr)
rather than passed along to grep.
--no-log: don't capture logs or run redo-log (same as pre-redo-log redo)
--no-pretty: don't pretty-print logs, print @@REDO lines.
The latter is an option to both redo and redo-log.
When running a parallel build, redo-log -f (which is auto-started by
redo) tries to traverse through the logs depth first, in the order
parent processes started subprocesses. This works pretty well, but if
its dependencies are locked, a process might have to give up its
jobserver token while other stuff builds its dependencies. After the
dependency finishes, the parent might not be able to get a token for
quite some time, and the logs will appear to stop.
To prevent this from happening, we can instantiate up to one "cheater"
token, only in the foreground process (the one locked by redo-log -f),
which will allow it to continue running, albeit a bit slowly (since it
only has one token out of possibly many). When the process finishes,
we then destroy the fake token. It gets a little complicated; see
explanation at the top of jwack.py.
