Merge branch 'py6' of https://github.com/mlell/redo
* 'py6' of https://github.com/mlell/redo:
Remove python<3.0 restriction in setup.py
Make compatible to BeautifulSoup4
Accept octal representations of Python 2 (0nnn) and Python 3 (0onnn)
Prevent iterator being changed while iterating
Python 2/3 compatible treatment of max(n, None)
Prevent "Exception ... ignored" in `redo-log ... | head`
Distinguish byte (python2 str type) and unicode strings (python 3 str type)
Set file descriptor as inheritable for all pythons >=3.4
Unify print function usage for Python 2 and 3 via __future__ import
Run 2to3 utility
Remove python interpreter selection
It's not reliable on all versions of docker, and we haven't had time to
fix it yet, so just turn it off. It's not essential to redo.
Reported-by: fcsmith@gmail.com
Open files in 'rb' (read binary) mode to prevent useless default utf8
encoding in python3, without breaking python2 compatibility.
Reported-by: Tharre <tharre3@gmail.com>
Some people have /usr/bin/python as a link to python3. The script is
designed to work in either python2 or python3, but python3's subprocess
model defaults to close_fds=True, which closes the jobserver fds and
causes an error. Explicitly force close_fds=False to get identical
behaviour on python2 and python3.
Reported-by: Tharre <tharre3@gmail.com>
This is a little simpler than the docs/cookbook/c project, which
doesn't actually have a doc yet because there was too much to explain.
I think I might make that a follow-on cookbook chapter, for people who
have read this simple one.
I think this doc is maybe a little too long; I intended it to be
"here's what you do to get started" but it turned into "here's what you
do to get started, and why it works, in excruciating detail." Not quite
sure how to fix.
(Also updated some other parts of the docs to refer to redoconf as a
real thing now instead of a "maybe someone should write this" thing.)
Apparently some newer docker versions crash if there is no "repository"
file in the resulting image. Its syntax seems simple enough, described
in https://github.com/moby/moby/blob/master/image/spec/v1.md
Reported-by: Matthew Singletary <matt.singletary@gmail.com>
Because redo targets are nicely isolated (unlike make targets), you
usually only want to debug one of them at a time. Using -x could be
confusing, because you might end up with a dump of output from a
dependency you're not interested in.
Now, by default we'll disable -x when recursing into sub-targets, so
you only see the trace from the targets you are actually trying to
debug. To get recursive behaviour, specify -x twice, eg. -xx.
Same idea with -v.
This lets us remove the awkward ". ./redoconf.rc" line from zillions
of .od scripts, without paying the price of *always* including that
whole file every time default.do.sh is used.
- libqt4.rc: add detection of the 'moc' command in $MOC.
- libssl.rc: add detection script for openssl.
- default.do.sh: provide a die() function to all .od files. It's just
too useful to not have it (I haven't converted everything to it yet).
- When building out/x/y.z, match against $S/default.z.od and
$S/default.od if they exist.
- *.list files (lists of source/object files to include in a binary)
can now contain *.o files in addition to *.c, *.cc, etc.
- rename compile() to _compile() to avoid polluting the namespace for
scripts called from default.do.sh.
- When building a .so file in the top level directory, no need to
make a symlink of it into the top level directory.
- link.od and link-shlib.od forgot to make use of the $xLIBS variable.
It was kind of ugly to have this kind of special flag directly in
link.od; now we detect whether it's available. This also gives a
project the ability to override the flag in some other way if they
want.
As a result, rc/shlib.rc needs to be included in all.rc.od if you want
to create or link against shared libraries.
While we're here, also add -Wl,-z,origin on platforms where it works.
This is apparently needed for FreeBSD 8.0 at least, and probably other
platforms.
Reported-by: Wayne Scott <wsc9tt@gmail.com>
Instead of running a bunch of separate rc_include statements in the
background, which causes unpredictable ordering of log output and
prevents -j from controlling parallelism, let's do a single
redo-ifchange for all of them (the slow part) followed by sequentially
checking the results (the fast part).
- Linking shared libraries needs slightly different options.
- We were trying to detect mach_time.h but needed to detect
mach/mach_time.h instead.
While we're here, add a --disable-shared option to ./configure, which
is different from --enable-static. --disable-shared does not build
*new* share libraries, but doesn't pass -static to the linker
(apparently there is no static linking posible on MacOS).
Now we define HAS_POSIX=1 if it works, and to blank otherwise, to be
consistent with other rc scripts, but it doesn't abort if we can't find
timespec.
Meanwhile, slightly clarify the error message in rc.sh.
Reported-by: Nathaniel Filardo <nwfilardo@gmail.com>
CC.rc was the only "mandatory" detection, which was weird and
inconsistent. Instead, make it optional like the others, and have it
set a HAVE_CC variable appropriately (and have CXX.rc work the same
way). Then, add a default.required.rc.od that checks the HAVE_* for
any variable and aborts if it is not available.
This allows us to fix confusing behaviour in allconfig.do, which would
try every compiler on the system, but redo would print a (non-fatal)
error message (and prevent redo-stamp optimization) when CC.rc failed
for any non-working compilers. Now CC.rc just politely reports that it
didn't find a compiler. Then we change all.rc.od to make CC.rc
mandatory.
Reported-by: Nathaniel Filardo <nwfilardo@gmail.com>
More generally, default.func.rc.od wasn't super useful because you
couldn't specify either a header file or library dependencies. Drop it
and make an explicit clock_gettime.rc.od. As a bonus, this also checks
that CLOCK_MONOTONIC exists as expected.
Reported-by: Wayne Scott <wsc9tt@gmail.com>
To test it out, try this:
./do -j10 build
cd docs/cookbook/c
redo -j10 test
It should detect all the compilers on your system and make three
separate builds for each one: normal, debug, and optimized. Then it
tries to run a test program under each one.
If there are windows cross compilers and you also have 'wine'
installed, it'll try running the test program under wine as well.
redoconf currently has no documentation other than the example program.
We'll fix that later.
If mkdocs is running in parallel, it can get upset if one of these
files exists when it lists the directory, but doesn't exist when it
goes to generate the output. Fundamentally this is a problem in mkdocs
more than in our code, but we might as well avoid it.
Silently recover if REDO_CHEATFDS file descriptors are closed, because
they aren't completely essential and MAKEFLAGS-related warnings already
get printed if all file descriptors have been closed.
If MAKEFLAGS --jobserver-auth flags are closed, improve the error
message so that a) it's a normal error instead of an exception and b)
we link to documentation about why it happens. Also write some more
detailed documentation about what's going on here.
These are often a good idea, but not necessary here and are distracting
to the tutorial, so let's just take them out.
Reported-by: Jeff Stearns <jeff.stearns@gmail.com>
This got... long... and complicated. But I think it's a really good
demonstration of getting redo to do complicated things elegantly. At
least, I hope it is.
We want to use the mkdocs-exclude plugin, which lets us exclude
particular files from the output directory. But plugins aren't
available in the debian-stable version of mkdocs, so ensure that we're
running a sufficiently new version. If we aren't, gracefully just skip
building the documentation.
all.do's main job was to print a "nothing much to do" message after
running. Nowadays it actually does do stuff, so we can remove the
warning, making _all.do redundant.
This avoids a name overlap with the system-installed copy of python.
Since redo adds the redo/ dir to the $PATH before running .do files,
python.do might see its own previously-created target instead of the
"real" python when testing, and create an infinite loop by accident.
It's time to start preparing for a version of redo that doesn't work
unless we build it first (because it will rely on C modules, and
eventually be rewritten in C altogether).
To get rolling, remove the old-style symlinks to the main programs, and
rename those programs from redo-*.py to redo/cmd_*.py. We'll also move
all library functions into the redo/ dir, which is a more python-style
naming convention.
Previously, install.do was generating wrappers for installing in
/usr/bin, which extend sys.path and then import+run the right file.
This made "installed" redo work quite differently from running redo
inside its source tree. Instead, let's always generate the wrappers in
bin/, and not make anything executable except those wrappers.
Since we're generating wrappers anyway, let's actually auto-detect the
right version of python for the running system; distros can't seem to
agree on what to call their python2 binaries (sigh). We'll fill in the
right #! shebang lines. Since we're doing that, we can stop using
/usr/bin/env, which will a) make things slightly faster, and b) let us
use "python -S", which tells python not to load a bunch of extra crap
we're not using, thus improving startup times.
Annoyingly, we now have to build redo using minimal/do, then run the
tests using bin/redo. To make this less annoying, we add a toplevel
./do script that knows the right steps, and a Makefile (whee!) for
people who are used to typing 'make' and 'make test' and 'make clean'.
