posh will abort the entire script if it detects a syntax error. I
don't know if that's good or not, but you shouldn't be writing scripts
with syntax errors, so that by itself isn't a good reason for posh to
fail.
It still fails some actual tests, but at least now we don't consider it
a 'crash' outcome.
If a file is overridden and then overridden again, this caused us to
rebuild only the first thing that depends on it, but not any subsequent
things, which is a pretty serious bug.
It turned out that t/350-deps-forget is already supposed to test this,
but I had cleverly encoded the wrong behaviour into the expected
results in the table-driven test. I blame lack of sleep. Anyway, I
fixed the test, which made it fail, and then fixed the code, which made
it pass.
Both redo and minimal/do were doing slightly weird things with
symlinked directories, especially when combined with "..". For
example, if x is a link to ., then x/x/x/x/../y should resolve to
"../y", which is quite non-obvious.
Added some tests to make sure this stays fixed.
Based on the earlier t/000-set-minus-e bug in minimal/do on some
shells, let's add some extra tests that reveal the weirdness on those
shells. Unfortunately because they are so popular (including bash and
zsh), we can't reject them outright for failing this one.
While we're here, add a new return code, "skip", which notes that a
test has failed but is not important enough to be considered a warning
or failure. Previously we just had these commented out, which is not
quite obvious enough.
...and I updated a few comments while reviewing some of the older
tests.
We were setting a global variable FAIL on failure, but if we failed
inside a subshell (which a very small number of tests might do), this
setting would be lost. The script output (a series of failed/warning
lines) was still valid, but not the return code, so the shell might be
selected even if one of these tests failed.
To avoid the problem, put the fail/warning state in the filesystem
instead, which is shared across subshells.
This can happen when $PWD contains a symlink somewhere in the path. In
that case, "cd ..; cat x" could mean something different from "cat ../x".
Notably, this error occurs when running "./do test" if your build
directory is through a symlink. For example, on freebsd your home
directory is /home/$USER, but /home is a symlink to /usr/home, which
triggers this problem.
Not adding tests in this commit, because when I added some tests, I
found even more symlink-related bugs, but those ones are much more
unlikely to occur. The additional fixes+tests are in a later commit.
Running commands in "||" context (like "x || return") disables "set -e"
behaviour in that context, even several levels deep in the call
hierarchy. The exact behaviour varies between shells, but this caused
a test failure with at least zsh 5.3.1 on debian.
Although I expect this is rather rare, some people may want to build in
a read-write subdir of a read-only tree. Other than some confusing
error reporting, this works fine in redo after the recent changes to
temp file handling, but let's add a test to make sure it stays that
way. The test found a bug in minimal/do, so let's fix that.
Reported-by: Jeff Stearns <jeff.stearns@gmail.com>
Previously, we'd try to put the stdout temp file in the same dir as the
target, if that dir exists. Otherwise we'd walk up the directory tree
looking for a good place. But this would go wrong if the directory we
chose got *deleted* during the run of the .do file.
Instead, we switch to an entirely new design: we use mkstemp() to
generate a temp file in the standard temp file location (probably
/tmp), then open it and immediately delete it, so the .do file can't
cause any unexpected behaviour. After the .do file exits, we use our
still-open fd to the stdout file to read the content back out.
In the old implementation, we also put the $3 in the "adjusted"
location that depended whether the target dir already existed, just for
consistency. But that was never necessary: we didn't create the $3
file, and if the .do script wants to write to $3, it should create the
target dir first anyway. So change it to *always* use a $3 temp
filename in the target dir, which is much simpler and so has fewer edge
cases.
Add t/202-del/deltest4 with some tests for all these edge cases.
Reported-by: Jeff Stearns <jeff.stearns@gmail.com>
This can happen if we create the .tmp file in the same directory as the
target, and the .do file first does "rm -rf" on that directory, then
re-creates it. The result is that the stdout file is lost.
We'll make this a warning if the .do script *didn't* write to stdout
(so the loss is harmless, just weird), and an error if they *did* write
to stdout, which we can detect because we still have an open fd on the
file, so we can fstat() it.
Basically all just missing quotes around shell strings that use $PWD.
Most paths inside a project, since redo uses relative paths, only need
to worry when project-internal directories or filenames have spaces in
them.
Reported-by: Jeff Stearns <jeff.stearns@gmail.com>
Now that the python scripts are all in a "redo" python module, we can
use the "new style" (ahem) package-relative imports. This appeases
pylint, plus avoids confusion in case more than one package has
similarly-named modules.
This removes another instance of magical code running at module import
time. And the process title wasn't really part of the state database
anyway.
Unfortunately this uncovered a bug: the recent change to use
'python -S' makes it not find the setproctitle module if installed.
My goodness, I hate the horrible python easy_install module gunk that
makes startup linearly slower the more modules you have installed,
whether you import them or not, if you don't use -S. But oh well,
we're stuck with it for now.
Merge the two files into env, and make each command explicitly call the
function that sets it up in the way that's needed for that command.
This means we can finally just import all the modules at the top of
each file, without worrying about import order. Phew.
While we're here, remove the weird auto-appending-'all'-to-targets
feature in env.init(). Instead, do it explicitly, and only from redo and
redo-ifchange, only if is_toplevel and no other targets are given.
Previously, we'd try to build all the critical stuff first, and then
run the tests. Nowadays, it takes a little longer to build the docs
(especially some of the docs/cookbook/ stuff), and this isn't needed to
run the tests, so let's allow them to parallelize.
This avoids a name overlap with the system-installed copy of python.
Since redo adds the redo/ dir to the $PATH before running .do files,
python.do might see its own previously-created target instead of the
"real" python when testing, and create an infinite loop by accident.
They really aren't locks at all, they're a cycle detector. Also rename
REDO_LOCKS to a more meaningful REDO_CYCLES. And we'll move the
CyclicDependencyError exception in here as well, instead of state.py
where it doesn't really belong.
It's time to start preparing for a version of redo that doesn't work
unless we build it first (because it will rely on C modules, and
eventually be rewritten in C altogether).
To get rolling, remove the old-style symlinks to the main programs, and
rename those programs from redo-*.py to redo/cmd_*.py. We'll also move
all library functions into the redo/ dir, which is a more python-style
naming convention.
Previously, install.do was generating wrappers for installing in
/usr/bin, which extend sys.path and then import+run the right file.
This made "installed" redo work quite differently from running redo
inside its source tree. Instead, let's always generate the wrappers in
bin/, and not make anything executable except those wrappers.
Since we're generating wrappers anyway, let's actually auto-detect the
right version of python for the running system; distros can't seem to
agree on what to call their python2 binaries (sigh). We'll fill in the
right #! shebang lines. Since we're doing that, we can stop using
/usr/bin/env, which will a) make things slightly faster, and b) let us
use "python -S", which tells python not to load a bunch of extra crap
we're not using, thus improving startup times.
Annoyingly, we now have to build redo using minimal/do, then run the
tests using bin/redo. To make this less annoying, we add a toplevel
./do script that knows the right steps, and a Makefile (whee!) for
people who are used to typing 'make' and 'make test' and 'make clean'.
Because our experiment involves creating a file called redo/sh, and the
redo directory (or some other version of redo's dirctory) might be on
the path, just testing 'sh' is actually wrong: we might end up with
some earlier version of redo-sh.
Instead, specifically for sh, always demand that we test /bin/sh.
This seems to only affect old zsh on MacOS. But we want to catch it
anyway, because it caused t/351-deps-forget to fail in a weird way on
that version of zsh.
Shells really suck.
Parallelism and redo-log cause lots of confusion for any rules that try
to ask the user for questions, so disable it altogether.
Arguably, we should just disable stdin all the time, but maybe it's
still occasionally useful (even though you have to pass --no-log to get
it back).
I attempted to fix this in commit c06d1fba40, but it was apparently
incomplete and not all cases were covered by tests.
Let's add a much more thorough test by going through every possible
combination and making sure redo-{sources,targets,ood} all work as
expected, that the "you modified it" warning does or does not show up
when expected, and that dependencies are rebuilt the number of times we
expect.
It seems like we're using these differently than most readthedocs.org
users. Remove the borders and padding so they work better inline, and
prevent confusing word wraps.
This shows how to dynamically generate a plot in R+ggplot2, then embed
it into a latex document, and compile it to pdf, all with proper
autodependencies.
Mailing list discussion was here:
https://groups.google.com/forum/#!topic/redo-list/wLMZMxtn4wo
Several more contributors replied to me personally to say that they
don't have a problem with the change (and several consider the change
to be an improvement).
The overwhelming majority of everything in the redo repo was written by
me, so I have the right to change the license unilaterally anyway,
subject to a few rules. Since the new license actually removes
licensing/usage restrictions for everyone, this should be no problem.
This was required for old versions of MacOS X, but the bug has now been
fixed, so let's de-clutter the .redo directory by reducing to a single
lock file instead of one per fid.
This reverts commit 08b05fd72f92fa0061b3d082b391151b43cd7497.
When we check dependencies and a previously-is_generated dependency
existed before, but no longer does, forget that it was is_generated.
This slightly improves the situation where as a project evolves, a file
that used to be a target gets removed, and then later is re-added as a
static source file. (It doesn't fix the other variant, where a file is
changed from target to source in a single atomic change, and is never
missing. That one will be trickier to handle.)
While adding a test for this behaviour, I discovered that redo-sources,
redo-targets, and redo-ood were reporting their output relative to
STARTDIR instead of relative to $PWD, so fix that too.
