- Linking shared libraries needs slightly different options.
- We were trying to detect mach_time.h but needed to detect
mach/mach_time.h instead.
While we're here, add a --disable-shared option to ./configure, which
is different from --enable-static. --disable-shared does not build
*new* share libraries, but doesn't pass -static to the linker
(apparently there is no static linking posible on MacOS).
Now we define HAS_POSIX=1 if it works, and to blank otherwise, to be
consistent with other rc scripts, but it doesn't abort if we can't find
timespec.
Meanwhile, slightly clarify the error message in rc.sh.
Reported-by: Nathaniel Filardo <nwfilardo@gmail.com>
CC.rc was the only "mandatory" detection, which was weird and
inconsistent. Instead, make it optional like the others, and have it
set a HAVE_CC variable appropriately (and have CXX.rc work the same
way). Then, add a default.required.rc.od that checks the HAVE_* for
any variable and aborts if it is not available.
This allows us to fix confusing behaviour in allconfig.do, which would
try every compiler on the system, but redo would print a (non-fatal)
error message (and prevent redo-stamp optimization) when CC.rc failed
for any non-working compilers. Now CC.rc just politely reports that it
didn't find a compiler. Then we change all.rc.od to make CC.rc
mandatory.
Reported-by: Nathaniel Filardo <nwfilardo@gmail.com>
More generally, default.func.rc.od wasn't super useful because you
couldn't specify either a header file or library dependencies. Drop it
and make an explicit clock_gettime.rc.od. As a bonus, this also checks
that CLOCK_MONOTONIC exists as expected.
Reported-by: Wayne Scott <wsc9tt@gmail.com>
To test it out, try this:
./do -j10 build
cd docs/cookbook/c
redo -j10 test
It should detect all the compilers on your system and make three
separate builds for each one: normal, debug, and optimized. Then it
tries to run a test program under each one.
If there are windows cross compilers and you also have 'wine'
installed, it'll try running the test program under wine as well.
redoconf currently has no documentation other than the example program.
We'll fix that later.
These files can appear/disappear during a parallel build, which
confuses mkdocs when it tries to refer to them, even though we never
wanted them in the docs anyway.
Reconnect the builder's original stderr file descriptor after the logger has
finished its job.
Fixes that redo could not be run without a controlling terminal.
We upgraded mkdocs earlier to support some useful features (like the
mkdocs-exclude plugin), but that one changes the highlight.js
configuration to include fewer languages by default. Specify the
additional ones we want to include.
Reported-by: @DRMacIver on twitter
If mkdocs is running in parallel, it can get upset if one of these
files exists when it lists the directory, but doesn't exist when it
goes to generate the output. Fundamentally this is a problem in mkdocs
more than in our code, but we might as well avoid it.
Silently recover if REDO_CHEATFDS file descriptors are closed, because
they aren't completely essential and MAKEFLAGS-related warnings already
get printed if all file descriptors have been closed.
If MAKEFLAGS --jobserver-auth flags are closed, improve the error
message so that a) it's a normal error instead of an exception and b)
we link to documentation about why it happens. Also write some more
detailed documentation about what's going on here.
These are often a good idea, but not necessary here and are distracting
to the tutorial, so let's just take them out.
Reported-by: Jeff Stearns <jeff.stearns@gmail.com>
This got... long... and complicated. But I think it's a really good
demonstration of getting redo to do complicated things elegantly. At
least, I hope it is.
We want to use the mkdocs-exclude plugin, which lets us exclude
particular files from the output directory. But plugins aren't
available in the debian-stable version of mkdocs, so ensure that we're
running a sufficiently new version. If we aren't, gracefully just skip
building the documentation.
WSL (Windows Services for Linux) provides a Linux-kernel-compatible ABI
for userspace processes, but the current version doesn't not implement
fcntl() locks at all; it just always returns success. See
https://github.com/Microsoft/WSL/issues/1927.
This causes us three kinds of problem:
1. sqlite3 in WAL mode gives "OperationalError: locking protocol".
1b. Other sqlite3 journal modes also don't work when used by
multiple processes.
2. redo parallelism doesn't work, because we can't prevent the same
target from being build several times simultaneously.
3. "redo-log -f" doesn't work, since it can't tell whether the log
file it's tailing is "done" or not.
To fix#1, we switch the sqlite3 journal back to PERSIST instead of
WAL. We originally changed to WAL in commit 5156feae9d to reduce
deadlocks on MacOS. That was never adequately explained, but PERSIST
still acts weird on MacOS, so we'll only switch to PERSIST when we
detect that locking is definitely broken. Sigh.
To (mostly) fix#2, we disable any -j value > 1 when locking is broken.
This prevents basic forms of parallelism, but doesn't stop you from
re-entrantly starting other instances of redo. To fix that properly,
we need to switch to a different locking mechanism entirely, which is
tough in python. flock() locks probably work, for example, but
python's locks lie and just use fcntl locks for those.
To fix#3, we always force --no-log mode when we find that locking is
broken.
On MacOS (at least 10.11.6), /bin/pwd defaults to using $PWD (ie. pwd
-L). On most other OSes it defaults to *not* using $PWD (ie. pwd -P).
We need the latter behaviour. It appears that 'pwd -P' has been
specified by POSIX for quite a few years now, so let's rely on it.
shelltest.od will now also check for it, though if your 'sh' doesn't
support this feature, it'll be too late, because shelltest needs
minimal/do in order to run.
This is unfixable when running with -j > 1 because of how the current
t/flush-cache script works. We'll only be able to fix that after
making a more granular flush-cache tool, which is already on my todo
list.
This new test validates that you can pass -j1 and -j2 in a sub-redo to
create a sub-jobserver with exactly the number of jobs you specified.
Now that we have that feature, we can also test for the bug fixed two
commits ago where, with -j1, targets would be built in an unexpected
order.
Previously, if you passed a -j option to a redo process in a redo or
make process hierarchy with MAKEFLAGS already set, it would ignore the
-j option and continue using the jobserver provided by the parent.
With this change, we instead initialize a new jobserver with the
desired number of tokens, which is what GNU make does in the same
situation. A typical use case for this is to force serialization of
build steps in a subtree (by using -j1). In make, this is often useful
for "fixing" makefiles that haven't been written correctly for parallel
builds. In redo, that happens much less often, but it's useful at
least in unit tests.
Passing -j1 is relatively harmless (the redo you are starting inherits
a token anyway, so it doesn't create any new tokens). Passing -j > 1
is more risky, because it creates new tokens, thus increasing the level
of parallelism in the system. Because this may not be what you wanted,
we print a warning when you pass -j > 1 to a sub-redo. GNU make gives
a similar warning in this situation.
After waiting for children to exit, we would release our own token, and
then the caller would immediately try to obtain a token again. This
accounted for tokens correctly, but would pass tokens around the call
tree in unexpected ways.
For example, imagine we had only one token. We call 'redo a1 a2', and
a1 calls 'redo b1 b2', and b1 calls 'redo c1'. When c1 exits, it
releases its token, then tries to re-acquire it before exiting. This
also includes 'redo b1 b2' and 'redo a1 a2' in the race for the token,
which means b1 might get suspended while *either* a2 or b2 starts
running.
This never caused a deadlock, even if a2 or b2 depends on b1, because
if they tried to build b1, they would notice it is locked, give up
their token, and wait for the lock. c1 (and then b1) could then obtain
the token and immediately terminate, allowing progress to continue.
But this is not really the way we expect things to happen. "Obviously"
what we want here is a straightforward stack unwinding: c1 should finish,
then b1, then b2, then a1, then b2.
The not-very-obvious symptom of this bug is that redo's unit tests
seemed to run in the wrong order when using -j1 --no-log. (--log would
hide the problem by rearranging logs back into the right order!)
all.do's main job was to print a "nothing much to do" message after
running. Nowadays it actually does do stuff, so we can remove the
warning, making _all.do redundant.
posh will abort the entire script if it detects a syntax error. I
don't know if that's good or not, but you shouldn't be writing scripts
with syntax errors, so that by itself isn't a good reason for posh to
fail.
It still fails some actual tests, but at least now we don't consider it
a 'crash' outcome.
If a file is overridden and then overridden again, this caused us to
rebuild only the first thing that depends on it, but not any subsequent
things, which is a pretty serious bug.
It turned out that t/350-deps-forget is already supposed to test this,
but I had cleverly encoded the wrong behaviour into the expected
results in the table-driven test. I blame lack of sleep. Anyway, I
fixed the test, which made it fail, and then fixed the code, which made
it pass.
Both redo and minimal/do were doing slightly weird things with
symlinked directories, especially when combined with "..". For
example, if x is a link to ., then x/x/x/x/../y should resolve to
"../y", which is quite non-obvious.
Added some tests to make sure this stays fixed.
Based on the earlier t/000-set-minus-e bug in minimal/do on some
shells, let's add some extra tests that reveal the weirdness on those
shells. Unfortunately because they are so popular (including bash and
zsh), we can't reject them outright for failing this one.
While we're here, add a new return code, "skip", which notes that a
test has failed but is not important enough to be considered a warning
or failure. Previously we just had these commented out, which is not
quite obvious enough.
...and I updated a few comments while reviewing some of the older
tests.
We were setting a global variable FAIL on failure, but if we failed
inside a subshell (which a very small number of tests might do), this
setting would be lost. The script output (a series of failed/warning
lines) was still valid, but not the return code, so the shell might be
selected even if one of these tests failed.
To avoid the problem, put the fail/warning state in the filesystem
instead, which is shared across subshells.
This can happen when $PWD contains a symlink somewhere in the path. In
that case, "cd ..; cat x" could mean something different from "cat ../x".
Notably, this error occurs when running "./do test" if your build
directory is through a symlink. For example, on freebsd your home
directory is /home/$USER, but /home is a symlink to /usr/home, which
triggers this problem.
Not adding tests in this commit, because when I added some tests, I
found even more symlink-related bugs, but those ones are much more
unlikely to occur. The additional fixes+tests are in a later commit.
Running commands in "||" context (like "x || return") disables "set -e"
behaviour in that context, even several levels deep in the call
hierarchy. The exact behaviour varies between shells, but this caused
a test failure with at least zsh 5.3.1 on debian.
Although I expect this is rather rare, some people may want to build in
a read-write subdir of a read-only tree. Other than some confusing
error reporting, this works fine in redo after the recent changes to
temp file handling, but let's add a test to make sure it stays that
way. The test found a bug in minimal/do, so let's fix that.
Reported-by: Jeff Stearns <jeff.stearns@gmail.com>
Previously, we'd try to put the stdout temp file in the same dir as the
target, if that dir exists. Otherwise we'd walk up the directory tree
looking for a good place. But this would go wrong if the directory we
chose got *deleted* during the run of the .do file.
Instead, we switch to an entirely new design: we use mkstemp() to
generate a temp file in the standard temp file location (probably
/tmp), then open it and immediately delete it, so the .do file can't
cause any unexpected behaviour. After the .do file exits, we use our
still-open fd to the stdout file to read the content back out.
In the old implementation, we also put the $3 in the "adjusted"
location that depended whether the target dir already existed, just for
consistency. But that was never necessary: we didn't create the $3
file, and if the .do script wants to write to $3, it should create the
target dir first anyway. So change it to *always* use a $3 temp
filename in the target dir, which is much simpler and so has fewer edge
cases.
Add t/202-del/deltest4 with some tests for all these edge cases.
Reported-by: Jeff Stearns <jeff.stearns@gmail.com>
This can happen if we create the .tmp file in the same directory as the
target, and the .do file first does "rm -rf" on that directory, then
re-creates it. The result is that the stdout file is lost.
We'll make this a warning if the .do script *didn't* write to stdout
(so the loss is harmless, just weird), and an error if they *did* write
to stdout, which we can detect because we still have an open fd on the
file, so we can fstat() it.
Basically all just missing quotes around shell strings that use $PWD.
Most paths inside a project, since redo uses relative paths, only need
to worry when project-internal directories or filenames have spaces in
them.
Reported-by: Jeff Stearns <jeff.stearns@gmail.com>
