diff --git a/Documentation/.gitignore b/Documentation/.gitignore new file mode 100644 index 0000000..ba94fa2 --- /dev/null +++ b/Documentation/.gitignore @@ -0,0 +1,2 @@ +/*.1 +/md-to-man diff --git a/Documentation/all.do b/Documentation/all.do new file mode 100644 index 0000000..b76c0c4 --- /dev/null +++ b/Documentation/all.do @@ -0,0 +1,5 @@ +/bin/ls *.md | +sed 's/\.md/.1/' | +xargs redo-ifchange + +redo-always diff --git a/Documentation/clean.do b/Documentation/clean.do new file mode 100644 index 0000000..2231500 --- /dev/null +++ b/Documentation/clean.do @@ -0,0 +1 @@ +rm -f *~ .*~ *.1 md-to-man diff --git a/Documentation/default.1.do b/Documentation/default.1.do new file mode 100644 index 0000000..26594b8 --- /dev/null +++ b/Documentation/default.1.do @@ -0,0 +1,2 @@ +redo-ifchange md-to-man $1.md +. ./md-to-man $1 $2 $3 diff --git a/Documentation/md-to-man.do b/Documentation/md-to-man.do new file mode 100644 index 0000000..8ea169a --- /dev/null +++ b/Documentation/md-to-man.do @@ -0,0 +1,6 @@ +if pandoc /dev/null; then + echo 'pandoc -s -r markdown -w man -o $3 $1.md' +else + echo "Warning: pandoc not installed; can't generate manpages." >&2 + echo 'echo Skipping: $1.1 >&2' +fi diff --git a/Documentation/redo-always.md b/Documentation/redo-always.md new file mode 100644 index 0000000..47e13cc --- /dev/null +++ b/Documentation/redo-always.md @@ -0,0 +1,53 @@ +% redo-always(1) Redo 0.00 +% Avery Pennarun +% 2010-12-12 + +# NAME + +redo-always - mark the current target as always needing to be rebuilt + +# SYNOPSIS + +redo-always + + +# DESCRIPTION + +Normally redo-always is run from a .do file that has been +executed by `redo`(1). See `redo`(1) for more details. + +redo-always takes no parameters. It simply adds an +'impossible' dependency to the current target, which +ensures that the target will always be rebuilt if anyone +runs `redo-ifchange targetname`. + +Because of the way redo works, `redo-ifchange targetname` +will only rebuild `targetname` once per session. So if +multiple targets depend on *targetname* and *targetname* +has called redo-always, only the first target will cause it +to be rebuilt. If the build cycle completes and a new one +begins, it will be rebuilt exactly one more time. + +Normally, any target that depends (directly or indirectly) +on a sub-target that has called redo-always will also +always need to rebuild, since one of its dependencies will +always be out of date. To avoid this problem, redo-always is +usually used along with `redo-stamp`(1). + + +# REDO + +Part of the `redo`(1) suite. + +# CREDITS + +The original concept for `redo` was created by D. J. +Bernstein and documented on his web site +(http://cr.yp.to/redo.html). This independent implementation +was created by Avery Pennarun and you can find its source +code at http://github.com/apenwarr/redo. + + +# SEE ALSO + +`redo`(1), `redo-ifcreate`(1), `redo-ifchange`(1), `redo-stamp`(1) diff --git a/Documentation/redo-ifchange.md b/Documentation/redo-ifchange.md new file mode 100644 index 0000000..7c4a71f --- /dev/null +++ b/Documentation/redo-ifchange.md @@ -0,0 +1,78 @@ +% redo-ifchange(1) Redo 0.00 +% Avery Pennarun +% 2010-12-12 + +# NAME + +redo-ifchange - rebuild target files when source files have changed + +# SYNOPSIS + +redo-ifchange [targets...] + + +# DESCRIPTION + +Normally redo-ifchange is run from a .do file that has been +executed by `redo`(1). See `redo`(1) for more details. + +redo-ifchange doesn't take any command line options other +than a list of *targets*. To provide command line options, +you need to run `redo` instead. + +redo-ifchange performs the following steps: + +- it creates a dependency on the given *targets*. If any + of those targets change in the future, the current target + (the one calling redo-ifchange) will marked as needing to + be rebuilt. + +- for any *target* that is out of date, it calls the + equivalent of `redo target`. + +- for any *target* that is locked (because some other + instance of `redo` or `redo-ifchange` is already building + it), it waits until the lock is released. + +redo-ifchange returns only after all the given +*targets* are known to be up to date. + + +# TIP + +You don't have to run redo-ifchange *before* generating +your target; you can generate your target first, then +declare its dependencies. For example, as part of +compiling a .c file, gcc learns the list +of .h files it depends on. You can pass this information +along to redo-ifchange, so if any of those headers are +changed or deleted, your .c file will be rebuilt: + + redo-ifchange $1$2 + gcc -o $3 -c $1$2 \ + -MMD -MF $1.deps -MT DELETE_ME + redo-ifchange $(sed -e 's,^DELETE_ME:,,' \ + -e 's,\\,,' $1.deps) + +This is much less confusing than the equivalent +autodependency mechanism in `make`(1), because make +requires that you declare all your dependencies before +running the target build commands. + + +# REDO + +Part of the `redo`(1) suite. + +# CREDITS + +The original concept for `redo` was created by D. J. +Bernstein and documented on his web site +(http://cr.yp.to/redo.html). This independent implementation +was created by Avery Pennarun and you can find its source +code at http://github.com/apenwarr/redo. + + +# SEE ALSO + +`redo`(1), `redo-ifcreate`(1), `redo-always`(1), `redo-stamp`(1) diff --git a/Documentation/redo-ifcreate.md b/Documentation/redo-ifcreate.md new file mode 100644 index 0000000..4a91672 --- /dev/null +++ b/Documentation/redo-ifcreate.md @@ -0,0 +1,47 @@ +% redo-ifcreate(1) Redo 0.00 +% Avery Pennarun +% 2010-12-12 + +# NAME + +redo-ifcreate - rebuild the current target if source files are created + +# SYNOPSIS + +redo-ifcreate [sources...] + + +# DESCRIPTION + +Normally redo-ifcreate is run from a .do file that has been +executed by `redo`(1). See `redo`(1) for more details. + +redo-ifcreate takes a list of nonexistent files (*sources*) +and adds them as dependencies to the current target (the +one calling redo-ifcreate). If any of those files are +created in the future, the target will be marked as needing +to be rebuilt. + +If one of the given files exists at the time redo-ifcreate +is called, it will return a nonzero exit code. + +If you want to declare dependencies on files that already +exist, use `redo-ifchange`(1) instead. + + +# REDO + +Part of the `redo`(1) suite. + +# CREDITS + +The original concept for `redo` was created by D. J. +Bernstein and documented on his web site +(http://cr.yp.to/redo.html). This independent implementation +was created by Avery Pennarun and you can find its source +code at http://github.com/apenwarr/redo. + + +# SEE ALSO + +`redo`(1), `redo-ifchange`(1), `redo-always`(1), `redo-stamp`(1) diff --git a/Documentation/redo-stamp.md b/Documentation/redo-stamp.md new file mode 100644 index 0000000..c965ad7 --- /dev/null +++ b/Documentation/redo-stamp.md @@ -0,0 +1,114 @@ +% redo-stamp(1) Redo 0.00 +% Avery Pennarun +% 2010-12-12 + +# NAME + +redo-stamp - detect if the current target has meaningfully changed + +# SYNOPSIS + +redo-stamp <\$3 + + +# DESCRIPTION + +Normally, when `redo`(1) builds a target T, all the other +targets that depend on T are marked as out of date. Even +if the rebuilt T is identical to the old one, all its +dependents need to be rebuilt. + +By calling redo-stamp from your .do script, you can tell +`redo` that even though the current target is building, its +output may turn out to be unchanged. If it hasn't, `redo` +may then be able to avoid building other targets that +depend on this target. + +redo-stamp marks the current target as changed or unchanged +by comparing its stdin to the input that was provided last +time redo-stamp was called for this target. + +The stamp data can be anything you want. Some possibilities +are: + +- the actual target file contents: + + redo-stamp <\$3 + +- a list of filenames: + + find -name '*.[ch]' | sort | redo-stamp + +- the contents of a web page: + + curl http://example.org | redo-stamp + +To ensure that your target gets checked every time, you +might want to use `redo-always`(1). + + +# DISCUSSION + +While using redo-stamp is simple, the way it +works is harder to explain. Deciding if a target is +up to date or not is the job of `redo-ifchange`(1). +Normally, a target is considered out of date when any of its +dependencies (direct or indirect) have changed. When that +happens, `redo-ifchange` runs the .do script for the +target, which regenerates the entire dependency list, +including rebuilding any dependencies as necessary. + +Imagine that we have the following dependency chain: + + T -> X -> Y -> Z + +T depends on X, which depends on Y, which depends +on Z. Now imagine someone has changed Z. + +If someone runs `redo-ifchange T`, then redo-ifchange +checks if X is up to date; to do that, it checks if Y +is up to date; and to do that, it checks whether Z is up to +date - which it isn't. Thus, Y is not up to date, which +means X isn't, which means T isn't either, and so we need +to run T.do. `redo-ifchange` won't directly `redo X` just +because X is dirty; it redoes T, and T.do might eventually +call `redo-ifchange X` if it needs to. + +When using redo-stamp, the behaviour is different. Let's +say Y has used redo-stamp. In the above sequence, Y is +outdated because Z has changed. However, we don't know yet +if Y's stamp has changed, so we don't yet know if we need +to redo X, and thus we don't know if we need to redo T. We +can't just run `redo T`, because that could waste a lot of +time if it turns out T didn't need to be rebuilt after all. + +What we do instead is note whether the *only* dependencies +of T that are out of date are 'stamped' targets. If *any* +dependencies of T are normal, out-of-date redo targets, +then T needs to be rebuilt anyway; we don't have to do +anything special. + +If the only dependency of T that has changed is Y (a +'stamped' target), then we need to `redo Y` automatically +in order to determine if T needs to be rebuilt. This is +the only time that `redo` ever rebuilds a target that +hasn't been explicitly asked for as part of a running .do +script. + + +# REDO + +Part of the `redo`(1) suite. + +# CREDITS + +The original concept for `redo` was created by D. J. +Bernstein and documented on his web site +(http://cr.yp.to/redo.html). This independent implementation +was created by Avery Pennarun and you can find its source +code at http://github.com/apenwarr/redo. + + +# SEE ALSO + +`redo`(1), `redo-ifcreate`(1), `redo-ifchange`(1), `redo-always`(1) diff --git a/Documentation/redo.md b/Documentation/redo.md new file mode 100644 index 0000000..e5e5451 --- /dev/null +++ b/Documentation/redo.md @@ -0,0 +1,223 @@ +% redo(1) Redo 0.00 +% Avery Pennarun +% 2010-12-12 + +# NAME + +redo - rebuild target files when source files have changed + +# SYNOPSIS + +redo [options...] [targets...] + + +# DESCRIPTION + +redo is a simple yet powerful tool for rebuilding target +files, and any of their dependencies, based on a set of +rules. The rules are encoded in simple `sh`(1) scripts +called '.do scripts.' + +redo supports GNU `make`(1)-style parallel builds using the +`-j` option; in fact, redo's parallel jobserver is compatible +with GNU Make, so redo and make can share build tokens with +each other. redo can call a sub-make (eg. to build a +subproject that uses Makefiles) or vice versa (eg. if a +make-based project needs to build a redo-based subproject). + +Unlike make, redo does not have any special syntax of its +own; each *target* is built by running a .do file, which is +simply a shell script that redo executes for you with a +particular environment and command-line arguments. + +If no *targets* are specified, redo pretends you specified +exactly one target named `all`. + +Note that redo *always* rebuilds the given targets +(although it may skip rebuilding the targets' dependencies +if they are up to date). If you only want to rebuild +targets that are not up to date, use `redo-ifchange`(1) +instead. + +A .do script can call redo recursively to build its +dependencies. + + +# OPTIONS + +-j, --jobs=*maxjobs* +: execute at most *maxjobs* .do scripts in parallel. The + default value is 1. + +-d, --debug +: print dependency checks as they happen. You can use + this to figure out why a particular target is/isn't being + rebuilt when your .do script calls it using + `redo-ifchange`. + +-v, --verbose +: pass the -v option to /bin/sh when executing scripts. + This normally causes the shell to echo the .do script lines + to stderr as it reads them. Most shells will print the + exact source line (eg. `echo $3`) and not the + substituted value of variables (eg. `echo + mytarget.redo.tmp`). + +-x, --xtrace +: pass the -x option to /bin/sh when executing scripts. + This normally causes the shell to echo exactly which + commands are being executed. Most shells will print + the substituted variables (eg. `echo + mytarget.redo.tmp`) and not the original source line + (eg. `echo $3`). + +-k, --keep-going +: keep building as many targets as possible even if some + of them return an error. If one target fails, any + target that depends on it also cannot be built, of course. + +--shuffle +: randomize the order in which requested targets are + built. Normally, if you run `redo a b c`, the targets + will be built exactly in that order: first `a`, then + `b`, then `c`. But if you use `-j`, they might end up + being built in parallel, so it isn't safe to rely on + this precise ordering. Using `--shuffle`, redo will + build its targets in random order even without `-j`, + which makes it easier to find accidental dependency + problems of this sort. NOTE: if you really just want + to guarantee that `a` is built, then `b`, then `c`, you + can just run three `redo` commands consecutively. + Because your .do script is just a script, it will not + be accidentally parallelized. + +--debug-locks +: print messages about acquiring, releasing, and waiting + on locks. Because redo can be highly parallelized, + one instance may end up waiting for a target to be + built by some other instance before it can continue. + If you suspect this is causing troubles, use this + option to see which instance is waiting and when. + +--debug-pids +: add the process id of the particular redo instance to each + output message. This makes it easier to figure out + which sub-instance of redo is doing what. + + +# DISCUSSION + +The core of redo is extremely simple. When you type `redo +targetname`, then it will search for a matching .do file +based on a simple algorithm. For example, given a target +named `mytarget.a.b.c.d`, redo will look for a .do file in +the following order: + +- mytarget.a.b.c.d.do +- default.a.b.c.d.do +- default.b.c.d.do +- default.c.d.do +- default.d.do +- default.do + +In all cases, the .do file must be in the same directory as +the target file. + +The first matching .do file is executed as a `/bin/sh` +script. The .do script is always executed with the current +working directory set to the directory containing the .do +file (and thus the target). Because of that rule, the +following two commands always have exactly identical +behaviour: + + redo path/to/target + + cd path/to && redo target + +(Note: in `make`(1), these commands have confusingly +different semantics. The first command would look for a +target named `path/to/target` in `./Makefile`, while the +second command would look for a target named `target` in +`./path/to/Makefile`. The two Makefiles might give +completely different results, and it's likely that the +first command would have incomplete dependency information. +redo does not have this problem.) + +The three arguments passed to the .do script are: + +- $1: the basename of the target (eg. mytarget.a.b) +- $2: the extension of the target, if any (eg. .c.d) +- $3: a temporary filename that the .do script should write + its output to. + +Instead of using $3, the .do script may also write the +produced data to stdout. + +redo is designed to update its targets atomically, and only +if the do script succeeds (ie. returns a zero exit code). +Thus, you should never write directly to the target file, +only to $3 or stdout. + +Normally, a .do script will call other .do scripts +recursively, by running either `redo` (which will always +build the sub-target) or `redo-ifchange` (which only +rebuilds the sub-target if its dependencies have changed). + +Running `redo-ifchange` is also the way your .do script +declares dependencies on other targets; any target that is +`redo-ifchange`d during your .do script's execution is both +executed (if needed) and added as a dependency. + +You may have heard that 'recursive make is considered +harmful' (http://miller.emu.id.au/pmiller/books/rmch/). +Unlike `make`(1), redo does correct locking, state +management, and global dependency checking, so none of the +arguments in that essay apply to redo. In fact, recursive +redo is really the only kind of redo. + + +# RELATED COMMANDS + +When writing a .do script, it will probably need to run +one or more of the following commands: + +`redo` +: to build a sub-target unconditionally. + +`redo-ifchange` +: to build a sub-target only if the sub-target's + dependencies have changed. + +`redo-ifcreate` +: to tell redo that the current target must be rebuilt if + a particular file gets created. + +`redo-always` +: to tell redo that the current target must always be + rebuilt, even if someone calls it using `redo-ifchange`. + (This might happen if the current target has + dependencies other than the contents of files.) + +`redo-stamp` +: to tell redo that even though the current target has + been rebuilt, it may not actually be any different from + the previous version, so targets that depend on it + might not need to be rebuilt. Often used in + conjunction with `redo-always` to reduce the impact of + always rebuilding a target. + + +# CREDITS + +The original concept for `redo` was created by D. J. +Bernstein and documented on his web site +(http://cr.yp.to/redo.html). This independent implementation +was created by Avery Pennarun and you can find its source +code at http://github.com/apenwarr/redo. + + +# SEE ALSO + +`sh`(1), `make`(1), +`redo-ifchange`(1), `redo-ifcreate`(1), `redo-always`(1), +`redo-stamp`(1) diff --git a/all.do b/all.do index 74e930d..340cd9f 100644 --- a/all.do +++ b/all.do @@ -1,2 +1,2 @@ -echo "Nothing to do. Try 'redo t/all' or 'redo test'" >&2 - +redo-ifchange Documentation/all +echo "Nothing much to do. Try 'redo t/all' or 'redo test'" >&2 diff --git a/clean.do b/clean.do index 71471ac..d88fdf2 100644 --- a/clean.do +++ b/clean.do @@ -4,7 +4,7 @@ if [ -e .do_built ]; then done <.do_built fi [ -z "$DO_BUILT" ] && rm -rf .do_built .do_built.dir -redo t/clean +redo t/clean Documentation/clean rm -f *~ .*~ */*~ */.*~ *.pyc rm -rf t/.redo find -name '*.tmp' -exec rm -fv {} \; diff --git a/test.do b/test.do index b5b0d6b..184db88 100644 --- a/test.do +++ b/test.do @@ -1,2 +1,2 @@ -redo-ifchange t/c.c +redo-ifchange Documentation/all t/c.c redo t/test