I must have changed this at the last minute when adding it, but I don't
know why. We were redefining f4() inside a subshell, so it never
applied to the parent shell, so it was always considered a failure.
But that's not what we were supposed to be testing.
This is just supposed to be a test of the really rare syntax of
defining a function without enclosing braces, which only fails on posh,
as far as I've seen.
If A calls B, and B produces stderr output, and then A wants to produce
output, the resulting log would be confusing: we'd see 'redo A' and
then 'redo B' and then B's output, but no indicator that B has ended
and we're back in A. Now we show 'redo A (resumed)' before A's output.
If B didn't produce any output, or A doesn't produce any output, we
don't bother with the (resumed) line. This seems nice, as it doesn't
clutter the log when there is no ambiguity anyway.
Because redo targets are nicely isolated (unlike make targets), you
usually only want to debug one of them at a time. Using -x could be
confusing, because you might end up with a dump of output from a
dependency you're not interested in.
Now, by default we'll disable -x when recursing into sub-targets, so
you only see the trace from the targets you are actually trying to
debug. To get recursive behaviour, specify -x twice, eg. -xx.
Same idea with -v.
This accidentally made output look different with --no-log vs with
normal redo-log output, because redo-log has a -u option while plain
redo does not.
(This is on purpose. When running redo, you only want to see the things
that actually happened, so it never passes -u to the auto-launched
redo-log instance. But when reviewing logs later, you might want to
look at the past logs from building a component that was unchanged in
the most recent run.)
* redoconf:
redoconf: a stub rc_include() now sources ./redoconf.rc automatically.
redoconf: assorted minor fixes.
redoconf: move -Wl,-rpath flags to shlib.rc and add -Wl,-z,origin.
mkdocs: don't bother to include cookbook/c/out.*/ dirs.
docs/cookbook/c/allconfig.do: avoid need for '&' backgrounding.
Precompiled headers: supply "-x c-header" or "-x c++-header" if available.
minimal/do: remove dependency on 'seq' command.
Fix some build problems on MacOS X.
redoconf: posix.rc: fix abort when timespec is not available.
redoconf: better handling of required vs optional detectors.
redoconf: clock_gettime() detection needs to depend on -lrt.
redoconf: avoid sed -E in _objlist().
Experimental new redoconf C/C++ build/autoconfiguration system.
This lets us remove the awkward ". ./redoconf.rc" line from zillions
of .od scripts, without paying the price of *always* including that
whole file every time default.do.sh is used.
- libqt4.rc: add detection of the 'moc' command in $MOC.
- libssl.rc: add detection script for openssl.
- default.do.sh: provide a die() function to all .od files. It's just
too useful to not have it (I haven't converted everything to it yet).
- When building out/x/y.z, match against $S/default.z.od and
$S/default.od if they exist.
- *.list files (lists of source/object files to include in a binary)
can now contain *.o files in addition to *.c, *.cc, etc.
- rename compile() to _compile() to avoid polluting the namespace for
scripts called from default.do.sh.
- When building a .so file in the top level directory, no need to
make a symlink of it into the top level directory.
- link.od and link-shlib.od forgot to make use of the $xLIBS variable.
* master:
builder.py: atomically replace the log for a given target.
redo-ifchange regression: if REDO_LOG is not set, assume it's 1.
Explicitly reject target/source filenames with newlines in them.
If redo searched all the way up to /default.do, it would run ./default.do instead.
Overridden files were accidentally getting reclassified as static.
Certain redo post-build failures would still mark a target as built.
minimal/do: remove dependency on 'seq' command.
Previously we were truncating the log if it existed. This would cause
redo-log to produce invalid output if you had the following (admittedly
rare) sequence in a single session:
- start building X
- redo-log starts showing the log for X
- finish building X
- redo-log has not finished showing the log for X yet
- start building X again for some reason
- redo-log sees a truncated logfile.
Now, redo-log can finish reading the original file (which no longer has
a filename since it was overwritten) while the new file is being
created.
At some point this got broken during a refactoring. The result was
that redo-ifchange, run from the command line (as opposed to inside a
.do script) would fail to start the log prettifier.
This only happened if the containing project was buggy, ie. you tried
to build a target that has no .do file available anywhere. However, it
resulted in a confusing outcome for that case, where we'd run the wrong
default.do file with the wrong parameters.
Extended an existing test to catch this mistake.
This is relatively harmless, since we treat them *almost* identically,
except that we print a warning for overridden files to remind you that
something fishy is going on.
Add a test for the actual warning message to ensure it is printed. (I
don't like tests for specific warning messages, but it was necessary in
this case.)
If we failed because:
- target dir doesn't exist
- failed to copy from stdout
- failed to rename $3
We would correctly return error 209, but the target would still be
marked as having been built, so redo-ifchange would not try to build it
next time, beause we forgot to call sf.set_failed() in those cases.
minimal/do worked correctly.
Added a test to catch this in the future.
It was kind of ugly to have this kind of special flag directly in
link.od; now we detect whether it's available. This also gives a
project the ability to override the flag in some other way if they
want.
As a result, rc/shlib.rc needs to be included in all.rc.od if you want
to create or link against shared libraries.
While we're here, also add -Wl,-z,origin on platforms where it works.
This is apparently needed for FreeBSD 8.0 at least, and probably other
platforms.
Reported-by: Wayne Scott <wsc9tt@gmail.com>
Instead of running a bunch of separate rc_include statements in the
background, which causes unpredictable ordering of log output and
prevents -j from controlling parallelism, let's do a single
redo-ifchange for all of them (the slow part) followed by sequentially
checking the results (the fast part).
Some older versions of gcc give "all.hpp: linker input file unused
because linking not done" otherwise, because they don't realize files
named *.hpp are headers.
Reported-by: Wayne Scott <wsc9tt@gmail.com>
- Linking shared libraries needs slightly different options.
- We were trying to detect mach_time.h but needed to detect
mach/mach_time.h instead.
While we're here, add a --disable-shared option to ./configure, which
is different from --enable-static. --disable-shared does not build
*new* share libraries, but doesn't pass -static to the linker
(apparently there is no static linking posible on MacOS).
Now we define HAS_POSIX=1 if it works, and to blank otherwise, to be
consistent with other rc scripts, but it doesn't abort if we can't find
timespec.
Meanwhile, slightly clarify the error message in rc.sh.
Reported-by: Nathaniel Filardo <nwfilardo@gmail.com>
CC.rc was the only "mandatory" detection, which was weird and
inconsistent. Instead, make it optional like the others, and have it
set a HAVE_CC variable appropriately (and have CXX.rc work the same
way). Then, add a default.required.rc.od that checks the HAVE_* for
any variable and aborts if it is not available.
This allows us to fix confusing behaviour in allconfig.do, which would
try every compiler on the system, but redo would print a (non-fatal)
error message (and prevent redo-stamp optimization) when CC.rc failed
for any non-working compilers. Now CC.rc just politely reports that it
didn't find a compiler. Then we change all.rc.od to make CC.rc
mandatory.
Reported-by: Nathaniel Filardo <nwfilardo@gmail.com>
More generally, default.func.rc.od wasn't super useful because you
couldn't specify either a header file or library dependencies. Drop it
and make an explicit clock_gettime.rc.od. As a bonus, this also checks
that CLOCK_MONOTONIC exists as expected.
Reported-by: Wayne Scott <wsc9tt@gmail.com>
To test it out, try this:
./do -j10 build
cd docs/cookbook/c
redo -j10 test
It should detect all the compilers on your system and make three
separate builds for each one: normal, debug, and optimized. Then it
tries to run a test program under each one.
If there are windows cross compilers and you also have 'wine'
installed, it'll try running the test program under wine as well.
redoconf currently has no documentation other than the example program.
We'll fix that later.
These files can appear/disappear during a parallel build, which
confuses mkdocs when it tries to refer to them, even though we never
wanted them in the docs anyway.
Reconnect the builder's original stderr file descriptor after the logger has
finished its job.
Fixes that redo could not be run without a controlling terminal.
We upgraded mkdocs earlier to support some useful features (like the
mkdocs-exclude plugin), but that one changes the highlight.js
configuration to include fewer languages by default. Specify the
additional ones we want to include.
Reported-by: @DRMacIver on twitter
If mkdocs is running in parallel, it can get upset if one of these
files exists when it lists the directory, but doesn't exist when it
goes to generate the output. Fundamentally this is a problem in mkdocs
more than in our code, but we might as well avoid it.
Silently recover if REDO_CHEATFDS file descriptors are closed, because
they aren't completely essential and MAKEFLAGS-related warnings already
get printed if all file descriptors have been closed.
If MAKEFLAGS --jobserver-auth flags are closed, improve the error
message so that a) it's a normal error instead of an exception and b)
we link to documentation about why it happens. Also write some more
detailed documentation about what's going on here.
These are often a good idea, but not necessary here and are distracting
to the tutorial, so let's just take them out.
Reported-by: Jeff Stearns <jeff.stearns@gmail.com>
This got... long... and complicated. But I think it's a really good
demonstration of getting redo to do complicated things elegantly. At
least, I hope it is.
We want to use the mkdocs-exclude plugin, which lets us exclude
particular files from the output directory. But plugins aren't
available in the debian-stable version of mkdocs, so ensure that we're
running a sufficiently new version. If we aren't, gracefully just skip
building the documentation.
WSL (Windows Services for Linux) provides a Linux-kernel-compatible ABI
for userspace processes, but the current version doesn't not implement
fcntl() locks at all; it just always returns success. See
https://github.com/Microsoft/WSL/issues/1927.
This causes us three kinds of problem:
1. sqlite3 in WAL mode gives "OperationalError: locking protocol".
1b. Other sqlite3 journal modes also don't work when used by
multiple processes.
2. redo parallelism doesn't work, because we can't prevent the same
target from being build several times simultaneously.
3. "redo-log -f" doesn't work, since it can't tell whether the log
file it's tailing is "done" or not.
To fix#1, we switch the sqlite3 journal back to PERSIST instead of
WAL. We originally changed to WAL in commit 5156feae9d to reduce
deadlocks on MacOS. That was never adequately explained, but PERSIST
still acts weird on MacOS, so we'll only switch to PERSIST when we
detect that locking is definitely broken. Sigh.
To (mostly) fix#2, we disable any -j value > 1 when locking is broken.
This prevents basic forms of parallelism, but doesn't stop you from
re-entrantly starting other instances of redo. To fix that properly,
we need to switch to a different locking mechanism entirely, which is
tough in python. flock() locks probably work, for example, but
python's locks lie and just use fcntl locks for those.
To fix#3, we always force --no-log mode when we find that locking is
broken.
On MacOS (at least 10.11.6), /bin/pwd defaults to using $PWD (ie. pwd
-L). On most other OSes it defaults to *not* using $PWD (ie. pwd -P).
We need the latter behaviour. It appears that 'pwd -P' has been
specified by POSIX for quite a few years now, so let's rely on it.
shelltest.od will now also check for it, though if your 'sh' doesn't
support this feature, it'll be too late, because shelltest needs
minimal/do in order to run.
This is unfixable when running with -j > 1 because of how the current
t/flush-cache script works. We'll only be able to fix that after
making a more granular flush-cache tool, which is already on my todo
list.
This new test validates that you can pass -j1 and -j2 in a sub-redo to
create a sub-jobserver with exactly the number of jobs you specified.
Now that we have that feature, we can also test for the bug fixed two
commits ago where, with -j1, targets would be built in an unexpected
order.
