forks/apenwarr-redo
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Directory reorg: move code into redo/, generate binaries in bin/.

Browse source 
It's time to start preparing for a version of redo that doesn't work
unless we build it first (because it will rely on C modules, and
eventually be rewritten in C altogether).

To get rolling, remove the old-style symlinks to the main programs, and
rename those programs from redo-*.py to redo/cmd_*.py.  We'll also move
all library functions into the redo/ dir, which is a more python-style
naming convention.

Previously, install.do was generating wrappers for installing in
/usr/bin, which extend sys.path and then import+run the right file.
This made "installed" redo work quite differently from running redo
inside its source tree.  Instead, let's always generate the wrappers in
bin/, and not make anything executable except those wrappers.

Since we're generating wrappers anyway, let's actually auto-detect the
right version of python for the running system; distros can't seem to
agree on what to call their python2 binaries (sigh). We'll fill in the
right #! shebang lines.  Since we're doing that, we can stop using
/usr/bin/env, which will a) make things slightly faster, and b) let us
use "python -S", which tells python not to load a bunch of extra crap
we're not using, thus improving startup times.

Annoyingly, we now have to build redo using minimal/do, then run the
tests using bin/redo.  To make this less annoying, we add a toplevel
./do script that knows the right steps, and a Makefile (whee!) for
people who are used to typing 'make' and 'make test' and 'make clean'.
This commit is contained in:
Avery Pennarun 2018-12-03 21:39:15 -05:00
commit f6fe00db5c
140 changed files with 256 additions and 99 deletions
Download patch file Download diff file Expand all files Collapse all files

5
docs/.gitignore vendored Normal file
View file

@ -0,0 +1,5 @@
*.1
/md-to-man
*.html
*.man.md
*.list

67
docs/Contributing.md Normal file
View file

@ -0,0 +1,67 @@
# License
My version of redo was written without ever seeing redo code by Bernstein or
Grosskurth, so I own the entire copyright. It's distributed under the terms
of the [Apache License 2.0](https://www.apache.org/licenses/LICENSE-2.0),
which you can find in the file called
[LICENSE](https://github.com/apenwarr/redo/tree/master/LICENSE). (Previous
versions of redo were distributed under the GNU LGPL, but this [changed in
2018](https://groups.google.com/forum/#!topic/redo-list/wLMZMxtn4wo).)
minimal/do is in the public domain so that it's as easy as possible
to include inside your own projects, to help end users who don't
have a copy of redo.
# How can I help?
Nowadays, redo is good enough for real production use, and some people
are using it for real work. That said, it has
not reached version 1.0 and there are surely still bugs.
If you run into a problem, it's really helpful if you report it to the
mailing list below (with or without subscribing first). We really want to
know if redo is acting weird for you. Even if the problem turns out to be
operator error, we can use that information to try to improve this
documentation.
Small feature additions are also welcome, but you might want to ask on the
mailing list before you start working on it. The code is still evolving and
might not be the same by the time you submit your pull request.
The best things you can do for redo are:
- Convert your projects to using it. Without users, no project is
successful.
- Build new infrastructure around redo, especially things to make it easier
for people to get started. For example, an automake-like tool that filled
in default redo build rules for common program types would probably be
very popular.
- Convince your favourite OS distro to build and include (up to date!) redo
packages.
- Help proofreading this documentation, and send patches or pull requests
with improvements.
- Join the mailing list (below) to discuss design changes or bugs.
- Tell your friends!
# Mailing list
You should join the `redo-list@googlegroups.com` mailing list.
You can find the mailing list archives here:
<http://groups.google.com/group/redo-list>
It might not look like it, but you can subscribe without having a
Google Account. Just send a message to
`redo-list+subscribe@googlegroups.com` (note the plus sign).
It's okay to send a message directly to the mailing list
without subscribing first. If you reply to someone who writes to the
list, please leave them in the cc: list, since if they
haven't subscribed, they won't get your reply otherwise.

143
docs/FAQBasics.md Normal file
View file

@ -0,0 +1,143 @@
# Help! redo rebuilds every time, even if dependencies are clean!
Many people are confused by this at first. The `redo` command *always*
rebuilds the target you requested. If you want to only conditionally
rebuild the target, use `redo-ifchange`.
The same rule applies inside .do scripts. If a .do script calls `redo dep`,
then it will always redo the target called dep. If it called `redo-ifchange
dep`, it will only redo dep if its dependencies have changed.
Typically, you want your toplevel `all.do` script to use `redo-ifchange`.
But a `clean.do` script probably should use `redo`, because you want it to
always clean up your extra files. A `test.do` probably also uses `redo` for
its testing sub-tasks, because you might want to run your tests repeatedly.
For rationale, see [Why does 'redo target' redo even unchanged
targets?](../FAQSemantics/#why-does-redo-target-redo-even-unchanged-targets)
# I'm using redo-ifchange. How does redo decide my target is clean/dirty?
You can see (recursively) which dependencies checked by redo, and whether
they are clean or dirty, by using `redo -d` (which is short for
"dependencies" or "debug", whichever you prefer).
# Does redo make cross-platform or multi-platform builds easy?
A lot of build systems that try to replace make do it by trying to provide a
lot of predefined rules. For example, one build system I know includes
default rules that can build C++ programs on Visual C++ or gcc,
cross-compiled or not cross-compiled, and so on. Other build systems are
specific to ruby programs, or python programs, or Java or .Net programs.
redo isn't like those systems; it's more like make. It doesn't know
anything about your system or the language your program is written in.
The good news is: redo will work with *any* programming language with about
equal difficulty. The bad news is: you might have to fill in more details
than you would if you just use ANT to compile a Java program.
So the short version is: cross-platform builds are about equally easy in
make and redo. It's not any easier, but it's not any harder.
It would be possible to make an automake-like or cmake-like tool that
generates .do files for your project, just like automake generates
Makefiles. But that's beyond the scope of redo itself.
# Can I set my dircolors to highlight .do files in ls output?
Yes! At first, having a bunch of .do files in each
directory feels like a bit of a nuisance, but once you get
used to it, it's actually pretty convenient; a simple 'ls'
will show you which things you might want to redo in any
given directory.
Here's a chunk of my .dircolors.conf:
.do 00;35
*Makefile 00;35
.o 00;30;1
.pyc 00;30;1
*~ 00;30;1
.tmp 00;30;1
To activate it, you can add a line like this to your .bashrc:
eval `dircolors $HOME/.dircolors.conf`
# Do end users have to have redo installed in order to build my project?
No. We include a very short and simple shell script
called `do` in the `minimal/` subdirectory of the redo project.
`minimal/do` is like
`redo` (and it works with the same `*.do` scripts), except it doesn't
understand dependencies; it just always rebuilds everything from the top.
You can include `do` with your program so that non-users of redo can
still build your program. Someone who wants to hack on your program will
probably go crazy unless they install a copy of `redo` though.
Actually, `redo` itself isn't so big, so for large projects where it
matters, you could just include it with your project.
# Recursive make is considered harmful. Isn't redo even *more* recursive?
You probably mean [this 1997 paper](http://miller.emu.id.au/pmiller/books/rmch/)
by Peter Miller.
Yes, redo is recursive, in the sense that every target is built by its own
`.do` file, and every `.do` file is a shell script being run recursively
from other shell scripts, which might call back into `redo`. In fact, it's
even more recursive than recursive make. There is no
non-recursive way to use redo.
However, the reason recursive make is considered harmful is that each
instance of make has no access to the dependency information seen by the
other instances. Each one starts from its own Makefile, which only has a
partial picture of what's going on; moreover, each one has to
stat() a lot of the same files over again, leading to slowness. That's
the thesis of the "considered harmful" paper.
It turns out that [non-recursive make should also be considered harmful](https://www.microsoft.com/en-us/research/wp-content/uploads/2016/03/hadrian.pdf).
The problem is Makefiles aren't
very "hygienic" or "modular"; if you're not running make recursively, then
your one copy of make has to know *everything* about *everything* in your
entire project. Every variable in make is global, so every variable defined
in *any* of your Makefiles is visible in *all* of your Makefiles. Every
little private function or macro is visible everywhere. In a huge project
made up of multiple projects from multiple vendors, that's just not okay.
Plus, if all your Makefiles are tangled together, make has
to read and parse the entire mess even to build the
smallest, simplest target file, making it slow.
`redo` deftly dodges both the problems of recursive make
and the problems of non-recursive make. First of all,
dependency information is shared through a global persistent `.redo`
database, which is accessed by all your `redo` instances at once.
Dependencies created or checked by one instance can be immediately used by
another instance. And there's locking to prevent two instances from
building the same target at the same time. So you get all the "global
dependency" knowledge of non-recursive make. And it's a
binary file, so you can just grab the dependency
information you need right now, rather than going through
everything linearly.
Also, every `.do` script is entirely hygienic and traceable; `redo`
discourages the use of global environment variables, suggesting that you put
settings into files (which can have timestamps and dependencies) instead.
So you also get all the hygiene and modularity advantages of recursive make.
By the way, you can trace any `redo` build process just by reading the `.do`
scripts from top to bottom. Makefiles are actually a collection of "rules"
whose order of execution is unclear; any rule might run at any time. In a
non-recursive Makefile setup with a bunch of included files, you end up with
lots and lots of rules that can all be executed in a random order; tracing
becomes impossible. Recursive make tries to compensate for this by breaking
the rules into subsections, but that ends up with all the "considered harmful"
paper's complaints. `redo` runs your scripts from top to bottom in a
nice tree, so it's traceable no matter how many layers you have.

371
docs/FAQImpl.md Normal file
View file

@ -0,0 +1,371 @@
# Does redo run on Windows?
redo works fine in [Windows Services for Linux (WSL)](https://docs.microsoft.com/en-us/windows/wsl/initialize-distro)
on Windows 10. You might consider that to be "real" Windows, or not. If
you use it, it runs more or less like it does on Linux. WSL is considerably
slower than native Linux, so there is definitely room for speed improvements.
If I were going to port redo to Windows in a "native" way,
I might grab the source code to a posix shell (like the
one in MSYS) and link it directly into redo.
One interesting project that has appeared recently is
busybox-w32 (https://github.com/pclouds/busybox-w32). It's
a port of busybox to win32 that includes a mostly POSIX
shell (ash) and a bunch of standard Unix utilities. This
might be enough to get your redo scripts working on a win32
platform without having to install a bunch of stuff. But
all of this needs more experimentation.
# Can a *.do file itself be generated as part of the build process?
Yes and no. Kind of. Redo doesn't stop you from doing this, and it will
use a .do file if you generate one. However, you have to do the steps in
the right order if you want this to work. For example, this should work
fine:
redo-ifchange default.o.do
redo-ifchange foo.o
You've told redo to generate `default.o.do` (presumably using a
script like default.do.do), and *then* you've told redo to generate `foo.o`.
When it considers how to build `foo.o`, it will look for a .do file, and
thus find `default.o.do`, so it'll run it. Great.
Some people would like us to go a step further, and *automatically* look for
rules that will help produce default.o.do. That is, you might want to just
write this:
redo-ifchange foo.o bar.o
and then expect that, implicitly, redo will know it needs to look for
`default.o.do`, and if `default.o.do` doesn't exist yet, it should look for
`default.do.do`, and so on.
The problem with this idea is... where does it end? If there's no
`default.o.do`, so we look for a `default.do.do`, what if that doesn't exist
either? Perhaps there's a `default.do.do.do` for generating `default.do.do`
files? And so on. You'd have to draw an arbitrary line somewhere.
Anyway, unlike make, redo does *not* implicitly generate files.
It only generates a given file if you redo-ifchange that file
first, from one of your other .do scripts. This gets avoids magical
behaviour, but makes it quite tricky to auto-generate .do files. See the
next question for a workaround.
# What can I do instead of auto-generating *.do files?
When people ask about auto-generating .do files, they usually mean one of
two things:
1. They want to create a new directory and auto-populate it with .do files
copied from somewhere.
- This can be solved in various ways. For example, you might make a
trivial toplevel `default.do` file in the new directory; this will
match all possible targets. You can then use the sh "source" operator
(.) and `redo-whichdo` to pull the "real" rules from elsewhere. An
example of [.do file delegation can be found in the wvstreams
project](https://github.com/apenwarr/wvstreams/blob/master/delegate.od).
2. They want to generate, eg. `default.o.do` based on auto-detected compiler
settings, in order to support different brands of compilers, or different
architectures, etc.
- The trick here is not to generate `default.o.do`, but rather to
generate another script instead. For example, you could have a
`compile.do` that generates a script called `compile`. `default.o.do`
would simply be hard coded to something like `./compile $2 $3`. The
wvstreams project also has [an example of
compile.do](https://github.com/apenwarr/wvstreams/blob/master/compile.do).
- The advantage of separating out into separate default.o.do and compile
scripts is that you can put all your system-dependent conditionals
into compile.do, so they run only once and choose the "right" compile
command. Then, for each file, you can simply run that command. If
This kind of micro-optimization doesn't appeal to you, then there's no
shame in just putting all the logic directly into `default.o.do`.
# How does redo store dependencies?
At the toplevel of your project, redo creates a directory
named `.redo`. That directory contains a sqlite3 database
with dependency information.
The format of the `.redo` directory is undocumented because
it may change at any time. It will likely turn out that we
can do something simpler than sqlite3. If you really need to make a
tool that pokes around in there, please ask on the mailing
list if we can standardize something for you.
Unfortunately, the design of having a *single* .redo directory at the
toplevel of a project has proven problematic: what exactly is the "top
level" of a "project"? If your project is a subdirectory of another project
that then switches to redo, should the .redo directory move up a level in
the hierarchy when that happens? And so on. Eventually, we will
probably migrate to a system where there is one .redo directory per target
directory. This avoids all kinds of problems with symlinks, directory
renames nested projects, and so on.
# Isn't using sqlite3 overkill? And un-djb-ish?
Well, yes. Sort of. I think people underestimate how
"lite" sqlite really is:
root root 573376 2010-10-20 09:55 /usr/lib/libsqlite3.so.0.8.6
573k for a *complete* and *very fast* and *transactional*
SQL database. For comparison, libdb is:
root root 1256548 2008-09-13 03:23 /usr/lib/libdb-4.6.so
...more than twice as big, and it doesn't even have an SQL parser in
it! Or if you want to be really horrified:
root root 1995612 2009-02-03 13:54 /usr/lib/libmysqlclient.so.15.0.0
The mysql *client* library is two megs, and it doesn't even
have a database in it! People who think SQL
databases are automatically bloated and gross have not yet
actually experienced the joys of sqlite. SQL has a
well-deserved bad reputation, but sqlite is another story
entirely. It's excellent, and much simpler and better
written than you'd expect.
Still, it's not very "djbish" to use a
general-purpose database, especially one that has a *SQL
parser* in it. (One of the great things about redo's
design is that it doesn't ever need to parse anything, so
embedding a whole SQL parser is a bit embarrassing.)
I'm pretty sure djb never would have done it that way.
However, I don't think we can reach the performance we want
with dependency/build/lock information stored in plain text
files; among other things, that results in too much
fstat/open activity, which is slow in general, and even
slower if you want to run on Windows. That leads us to a
binary database. And example of the kind of structure we need is the [one
used by ninja](https://github.com/ninja-build/ninja/blob/master/src/deps_log.h#L29)
which is very simple, fast, and efficient.
Most of the state database stuff has been isolated in
state.py. If you're feeling brave, you can try to
implement your own better state database, with or without
sqlite.
# What hash algorithm does redo-stamp use?
It's intentionally undocumented because you shouldn't need
to care and it might change at any time. But trust me,
it's not the slow part of your build, and you'll never
accidentally get a hash collision.
# Why not *always* use checksum-based dependencies instead of timestamps?
Some build systems keep a checksum of target files and rebuild dependents
only when the target changes. This is appealing in some cases; for example,
with ./configure generating config.h, it could just go ahead and generate
config.h; the build system would be smart enough to rebuild or not rebuild
dependencies automatically. This keeps build scripts simple and gets rid of
the need for people to re-implement file comparison over and over in every
project or for multiple files in the same project.
There are disadvantages to using checksums for everything
automatically, however:
- Building stuff unnecessarily is *much* less dangerous
than not building stuff that should be built. Checksums
aren't perfect (think of zero-byte output files); using
checksums will cause more builds to be skipped by
default, which is very dangerous.
- It makes it hard to *force* things to rebuild when you
know you absolutely want that. (With timestamps, you can
just `touch filename` to rebuild everything that depends
on `filename`.)
- Targets that are just used for aggregation (ie. they
don't produce any output of their own) would always have
the same checksum - the checksum of a zero-byte file -
which causes confusing results.
- Calculating checksums for every output file adds time to
the build, even if you don't need that feature.
- Building stuff unnecessarily and then stamping it is
much slower than just not building it in the first place,
so for *almost* every use of redo-stamp, it's not the
right solution anyway.
- To steal a line from the Zen of Python: explicit is
better than implicit. Making people think about when
they're using the stamp feature - knowing that it's slow
and a little annoying to do - will help people design
better build scripts that depend on this feature as
little as possible.
- djb's (as yet unreleased) version of redo doesn't
implement checksums, so doing that would produce an
incompatible implementation. With redo-stamp and
redo-always being separate programs, you can simply
choose not to use them if you want to keep maximum
compatibility for the future.
- Bonus: the redo-stamp algorithm is interchangeable. You
don't have to stamp the target file or the source files
or anything in particular; you can stamp any data you
want, including the output of `ls` or the content of a
web page. We could never have made things like that
implicit anyway, so some form of explicit redo-stamp
would always have been needed, and then we'd have to
explain when to use the explicit one and when to use the
implicit one.
Thus, we made the decision to only use checksums for
targets that explicitly call `redo-stamp` (see previous
question).
I suggest actually trying it out to see how it feels for
you. For myself, before there was redo-stamp and
redo-always, a few types of problems (in particular,
depending on a list of which files exist and which don't)
were really annoying, and I definitely felt it. Adding
redo-stamp and redo-always work the way they do made the
pain disappear, so I stopped changing things.
A longer and even more detailed explanation of timestamp vs checksum-based
build dependencies can be found in
[mtime comparison considered harmful](https://apenwarr.ca/log/20181113).
# Why doesn't redo by default print the commands as they are run?
make prints the commands it runs as it runs them. redo doesn't, although
you can get this behaviour with `redo -v` or `redo -x`.
(The difference between -v and -x is the same as it is in
sh... because we simply forward those options onward to sh
as it runs your .do script.)
The main reason we don't do this by default is that the commands get
pretty long
winded (a compiler command line might be multiple lines of repeated
gibberish) and, on large projects, it's hard to actually see the progress of
the overall build. Thus, make users often work hard to have make hide the
command output in order to make the log "more readable."
The reduced output is a pain with make, however, because if there's ever a
problem, you're left wondering exactly what commands were run at what time,
and you often have to go editing the Makefile in order to figure it out.
With redo, it's much less of a problem. By default, redo produces output
that looks like this:
$ redo t
redo t/all
redo t/hello
redo t/LD
redo t/hello.o
redo t/CC
redo t/yellow
redo t/yellow.o
redo t/bellow
redo t/c
redo t/c.c
redo t/c.c.c
redo t/c.c.c.b
redo t/c.c.c.b.b
redo t/d
The indentation indicates the level of recursion (deeper levels are
dependencies of earlier levels). The repeated word "redo" down the left
column looks strange, but it's there for a reason, and the reason is this:
you can cut-and-paste a line from the build script and rerun it directly.
$ redo t/c
redo t/c
redo t/c.c
redo t/c.c.c
redo t/c.c.c.b
redo t/c.c.c.b.b
So if you ever want to debug what happened at a particular step, you can
choose to run only that step in verbose mode:
$ redo t/c.c.c.b.b -x
redo t/c.c.c.b.b
* sh -ex default.b.do c.c.c.b .b c.c.c.b.b.redo2.tmp
+ redo-ifchange c.c.c.b.b.a
+ echo a-to-b
+ cat c.c.c.b.b.a
+ ./sleep 1.1
redo t/c.c.c.b.b (done)
If you're using an autobuilder or something that logs build results for
future examination, you should probably set it to always run redo with
the -x option.
# How fast is redo compared to make?
FIXME:
The current version of redo is written in python and has not been optimized.
So right now, it's usually a bit slower. Not too embarrassingly slower,
though, and the slowness mostly only strikes when you're
building a project from scratch.
For incrementally building only the changed parts of the project, redo can
be much faster than make, because it can check all the dependencies up
front and doesn't need to repeatedly parse and re-parse the Makefile (as
recursive make needs to do).
redo's sqlite3-based dependency database is very fast (and
it would be even faster if we rewrite redo in C instead of
python). Better still, it would be possible to write an
inotify daemon that can update the dependency database in
real time; if you're running the daemon, you can run 'redo'
from the toplevel and if your build is clean, it could return
instantly, no matter how many dependencies you have.
On my machine, redo can currently check about 10,000
dependencies per second. As an example, a program that
depends on every single .c or .h file in the Linux kernel
2.6.36 repo (about 36000 files) can be checked in about 4
seconds.
Rewritten in C, dependency checking would probably go about
10 times faster still.
This probably isn't too hard; the design of redo is so simple that
it should be easy to write in any language. It's just
*even easier* in python, which was good for writing the
prototype and debugging the parallelism and locking rules.
Most of the slowness at the moment is because redo-ifchange
(and also sh itself) need to be fork'd and exec'd over and
over during the build process.
As a point of reference, on my computer, I can fork-exec
redo-ifchange.py about 87 times per second; an empty python
program, about 100 times per second; an empty C program,
about 1000 times per second; an empty make, about 300 times
per second. So if I could compile 87 files per second with
gcc, which I can't because gcc is slower than that, then
python overhead would be 50%. Since gcc is slower than
that, python overhead is generally much less - more like
10%.
Also, if you're using redo -j on a multicore machine, all
the python forking happens in parallel with everything
else, so that's 87 per second per core. Nevertheless,
that's still slower than make and should be fixed.
(On the other hand, all this measurement is confounded
because redo's more fine-grained dependencies mean you can
have more parallelism. So if you have a lot of CPU cores, redo
might build *faster* than make just because it makes better
use of them.)

55
docs/FAQInterop.md Normal file
View file

@ -0,0 +1,55 @@
# Is redo compatible with autoconf?
Yes. You don't have to do anything special, other than the above note about
declaring dependencies on config.h, which is no worse than what you would
have to do with make.
# Is redo compatible with automake?
Hells no. You can thank me later. But see next question.
# Is redo compatible with make?
Yes. If you have an existing Makefile (for example, in one of your
subprojects), you can just call make from a .do script to build that
subproject.
In a file called myproject.stamp.do:
redo-ifchange $(find myproject -name '*.[ch]')
make -C myproject all
So, to amend our answer to the previous question, you *can* use
automake-generated Makefiles as part of your redo-based project.
# Is redo -j compatible with make -j?
Yes! redo implements the same jobserver protocol as GNU make, which means
that redo running under make -j, or make running under redo -j, will do the
right thing. Thus, it's safe to mix-and-match redo and make in a recursive
build system.
Just make sure you declare your dependencies correctly;
redo won't know all the specific dependencies included in
your Makefile, and make won't know your redo dependencies,
of course.
One way of cheating is to just have your make.do script
depend on *all* the source files of a subproject, like
this:
make -C subproject all
find subproject -name '*.[ch]' | xargs redo-ifchange
Now if any of the .c or .h files in subproject are changed,
your make.do will run, which calls into the subproject to
rebuild anything that might be needed. Worst case, if the
dependencies are too generous, we end up calling 'make all'
more often than necessary. But 'make all' probably runs
pretty fast when there's nothing to do, so that's not so
bad.

158
docs/FAQParallel.md Normal file
View file

@ -0,0 +1,158 @@
# Parallelism if more than one target depends on the same subdir
Recursive make is especially painful when it comes to
parallelism. Take a look at this Makefile fragment:
all: fred bob
subproj:
touch $@.new
sleep 1
mv $@.new $@
fred:
$(MAKE) subproj
touch $@
bob:
$(MAKE) subproj
touch $@
If we run it serially, it all looks good:
$ rm -f subproj fred bob; make --no-print-directory
make subproj
touch subproj.new
sleep 1
mv subproj.new subproj
touch fred
make subproj
make[1]: 'subproj' is up to date.
touch bob
But if we run it in parallel, life sucks:
$ rm -f subproj fred bob; make -j2 --no-print-directory
make subproj
make subproj
touch subproj.new
touch subproj.new
sleep 1
sleep 1
mv subproj.new subproj
mv subproj.new subproj
mv: cannot stat 'ubproj.new': No such file or directory
touch fred
make[1]: *** [subproj] Error 1
make: *** [bob] Error 2
What happened? The sub-make that runs `subproj` ended up
getting twice at once, because both fred and bob need to
build it.
If fred and bob had put in a *dependency* on subproj, then
GNU make would be smart enough to only build one of them at
a time; it can do ordering inside a single make process.
So this example is a bit contrived. But imagine that fred
and bob are two separate applications being built from the
same toplevel Makefile, and they both depend on the library
in subproj. You'd run into this problem if you use
recursive make.
Of course, you might try to solve this by using
*nonrecursive* make, but that's really hard. What if
subproj is a library from some other vendor? Will you
modify all their makefiles to fit into your nonrecursive
makefile scheme? Probably not.
Another common workaround is to have the toplevel Makefile
build subproj, then fred and bob. This works, but if you
don't run the toplevel Makefile and want to go straight
to work in the fred project, building fred won't actually
build subproj first, and you'll get errors.
redo solves all these problems. It maintains global locks
across all its instances, so you're guaranteed that no two
instances will try to build subproj at the same time. And
this works even if subproj is a make-based project; you
just need a simple subproj.do that runs `make subproj`.
# Dependency problems that only show up during parallel builds
One annoying thing about parallel builds is... they do more
things in parallel. A very common problem in make is to
have a Makefile rule that looks like this:
all: a b c
When you `make all`, it first builds a, then b, then c.
What if c depends on b? Well, it doesn't matter when
you're building in serial. But with -j3, you end up
building a, b, and c at the same time, and the build for c
crashes. You *should* have said:
all: a b c
c: b
b: a
and that would have fixed it. But you forgot, and you
don't find out until you build with exactly the wrong -j
option.
This mistake is easy to make in redo too. But it does have
a tool that helps you debug it: the --shuffle option.
--shuffle takes the dependencies of each target, and builds
them in a random order. So you can get parallel-like
results without actually building in parallel.
# What about distributed builds?
FIXME:
So far, nobody has tried redo in a distributed build environment. It surely
works with distcc, since that's just a distributed compiler. But there are
other systems that distribute more of the build process to other machines.
The most interesting method I've heard of was explained (in public, this is
not proprietary information) by someone from Google. Apparently, the
Android team uses a tool that mounts your entire local filesystem on a
remote machine using FUSE and chroots into that directory. Then you replace
the $SHELL variable in your copy of make with one that runs this tool.
Because the remote filesystem is identical to yours, the build will
certainly complete successfully. After the $SHELL program exits, the changed
files are sent back to your local machine. Cleverly, the files on the
remote server are cached based on their checksums, so files only need to be
re-sent if they have changed since last time. This dramatically reduces
bandwidth usage compared to, say, distcc (which mostly just re-sends the
same preparsed headers over and over again).
At the time, he promised to open source this tool eventually. It would be
pretty fun to play with it.
The problem:
This idea won't work as easily with redo as it did with
make. With make, a separate copy of $SHELL is launched for
each step of the build (and gets migrated to the remote
machine), but make runs only on your local machine, so it
can control parallelism and avoid building the same target
from multiple machines, and so on. The key to the above
distribution mechanism is it can send files to the remote
machine at the beginning of the $SHELL, and send them back
when the $SHELL exits, and know that nobody cares about
them in the meantime. With redo, since the entire script
runs inside a shell (and the shell might not exit until the
very end of the build), we'd have to do the parallelism
some other way.
I'm sure it's doable, however. One nice thing about redo
is that the source code is so small compared to make: you
can just rewrite it.
# Can I convince a sub-redo or sub-make to *not* use parallel builds?
Yes. Put this in your .do script:
unset MAKEFLAGS
The child makes will then not have access to the jobserver,
so will build serially instead.

582
docs/FAQSemantics.md Normal file
View file

@ -0,0 +1,582 @@
# Can I put all my rules in one big Redofile like make does?
One of my favourite features of redo is that it doesn't add any new syntax;
the syntax of redo is *exactly* the syntax of sh... because sh is the program
interpreting your .do file.
Also, it's surprisingly useful to have each build script in its own file;
that way, you can declare a dependency on just that one build script instead
of the entire Makefile, and you won't have to rebuild everything just
because of a one-line Makefile change. (Some build tools avoid that same
problem by tracking which variables and commands were used to do the build.
But that's more complex, more error prone, and slower.)
See djb's [Target files depend on build scripts](http://cr.yp.to/redo/honest-script.html)
article for more information.
However, if you really want to, you can simply create a
default.do that looks something like this:
case $1 in
*.o) ...compile a .o file... ;;
myprog) ...link a program... ;;
*) echo "no rule to build '$1'" >&2; exit 1 ;;
esac
Basically, default.do is the equivalent of a central
Makefile in make. As of recent versions of redo, you can
use either a single toplevel default.do (which catches
requests for files anywhere in the project that don't have
their own .do files) or one per directory, or any
combination of the above. And you can put some of your
targets in default.do and some of them in their own files.
Lay it out in whatever way makes sense to you.
One more thing: if you put all your build rules in a single
default.do, you'll soon discover that changing *anything*
in that default.do will cause all your targets to rebuilt -
because their .do file has changed. This is technically
correct, but you might find it annoying. To work around
it, try making your default.do look like this:
. ./default.od
And then put the above case statement in default.od
instead. Since you didn't `redo-ifchange default.od`,
changes to default.od won't cause everything to rebuild.
# What are the parameters ($1, $2, $3) to a .do file?
NOTE: These definitions have changed since the earliest
(pre-0.10) versions of redo. The new definitions match
what djb's original redo implementation did.
$1 is the name of the target file.
$2 is the basename of the target, minus the extension, if
any.
$3 is the name of a temporary file that will be renamed to
the target filename atomically if your .do file returns a
zero (success) exit code.
In a file called `chicken.a.b.c.do` that builds a file called
`chicken.a.b.c`, $1 and $2 are `chicken.a.b.c`, and $3 is a
temporary name like `chicken.a.b.c.tmp`. You might have expected
$2 to be just `chicken`, but that's not possible, because
redo doesn't know which portion of the filename is the
"extension." Is it `.c`, `.b.c`, or `.a.b.c`?
.do files starting with `default.` are special; they can
build any target ending with the given extension. So let's
say we have a file named `default.c.do` building a file
called `chicken.a.b.c`. $1 is `chicken.a.b.c`, $2 is `chicken.a.b`,
and $3 is a temporary name like `chicken.a.b.c.tmp`.
You should use $1 and $2 only in constructing input
filenames and dependencies; never modify the file named by
$1 in your script. Only ever write to the file named by
$3. That way redo can guarantee proper dependency
management and atomicity. (For convenience, you can write
to stdout instead of $3 if you want.)
For example, you could compile a .c file into a .o file
like this, from a script named `default.o.do`:
redo-ifchange $2.c
gcc -o $3 -c $2.c
# Why not $FILE, $BASE, $OUT instead of $1, $2, $3?
That sounds tempting and easy, but one downside would be
lack of backward compatibility with djb's original redo
design.
Longer names aren't necessarily better. Learning the
meanings of the three numbers doesn't take long, and over
time, those extra few keystrokes can add up. And remember
that Makefiles and perl have had strange one-character
variable names for a long time. It's not at all clear that
removing them is an improvement.
# What happens to stdin/stdout/stderr?
As with make, stdin is not redirected. You're probably
better off not using it, though, because especially with
parallel builds, it might not do anything useful. We might
change this behaviour someday since it's such a terrible
idea for .do scripts to read from stdin.
As with make, stderr is also not redirected. You can use
it to print status messages as your build proceeds.
(Eventually, we might want to capture stderr so it's easier
to look at the results of parallel builds, but this is
tricky to do in a user-friendly way.)
Redo treats stdout specially: it redirects it to point at
$3 (see previous question). That is, if your .do file
writes to stdout, then the data it writes ends up in the
output file. Thus, a really simple `chicken.do` file that
contains only this:
echo hello world
will correctly, and atomically, generate an output file
named `chicken` only if the echo command succeeds.
# Isn't it confusing to capture stdout by default?
Yes, it is. It's unlike what almost any other program
does, especially make, and it's very easy to make a
mistake. For example, if you write in your script:
echo "Hello world"
it will go to the target file rather than to the screen.
A more common mistake is to run a program that writes to
stdout by accident as it runs. When you do that, you'll
produce your target on $3, but it might be intermingled
with junk you wrote to stdout. redo is pretty good about
catching this mistake, and it'll print a message like this:
redo zot.do wrote to stdout *and* created $3.
redo ...you should write status messages to stderr, not stdout.
redo zot: exit code 207
Despite the disadvantages, though, automatically capturing
stdout does make certain kinds of .do scripts really
elegant. The "simplest possible .do file" can be very
short. For example, here's one that produces a sub-list
from a list:
redo-ifchange filelist
grep ^src/ filelist
redo's simplicity is an attempt to capture the "Zen of
Unix," which has a lot to do with concepts like pipelines
and stdout. Why should every program have to implement its
own -o (output filename) option when the shell already has
a redirection operator? Maybe if redo gets more popular,
more programs in the world will be able to be even simpler
than they are today.
By the way, if you're running some programs that might
misbehave and write garbage to stdout instead of stderr
(Informational/status messages always belong on stderr, not
stdout! Fix your programs!), then just add this line to
the top of your .do script:
exec >&2
That will redirect your stdout to stderr, so it works more
like you expect.
# Run redo-ifchange in a loop?
The obvious way to write a list of dependencies might be
something like this:
for d in *.c; do
redo-ifchange ${d%.c}.o
done
But it turns out that's very non-optimal. First of all, it
forces all your dependencies to be built in order
(redo-ifchange doesn't return until it has finished
building), which makes -j parallelism a lot less useful.
And secondly, it forks and execs redo-ifchange over and
over, which can waste CPU time unnecessarily.
A better way is something like this:
for d in *.c; do
echo ${d%.c}.o
done |
xargs redo-ifchange
That only runs redo-ifchange once (or maybe a few times, if
there are really a *lot* of dependencies and xargs has to
split it up), which saves fork/exec time and allows for
parallelism.
# If a target is identical after rebuilding, how do I prevent dependents from being rebuilt?
For example, running ./configure creates a bunch of files including
config.h, and config.h might or might not change from one run to the next.
We don't want to rebuild everything that depends on config.h if config.h is
identical.
With `make`, which makes build decisions based on timestamps, you would
simply have the ./configure script write to config.h.new, then only
overwrite config.h with that if the two files are different.
However, that's a bit tedious.
With `redo`, there's an easier way. You can have a
config.do script that looks like this:
redo-ifchange autogen.sh *.ac
./autogen.sh
./configure
cat config.h configure Makefile | redo-stamp
Now any of your other .do files can depend on a target called
`config`. `config` gets rebuilt automatically if any of
your autoconf input files are changed (or if someone does
`redo config` to force it). But because of the call to
redo-stamp, `config` is only considered to have changed if
the contents of config.h, configure, or Makefile are
different than they were before.
(Note that you might actually want to break this .do up into a
few phases: for example, one that runs aclocal, one that
runs autoconf, and one that runs ./configure. That way
your build can always do the minimum amount of work
necessary.)
# Why does 'redo target' redo even unchanged targets?
When you run `make target`, make first checks the
dependencies of target; if they've changed, then it
rebuilds target. Otherwise it does nothing.
redo is a little different. It splits the build into two
steps. `redo target` is the second step; if you run that
at the command line, it just runs the .do file, whether it
needs it or not.
If you really want to only rebuild targets that have
changed, you can run `redo-ifchange target` instead.
The reasons I like this arrangement come down to semantics:
- "make target" implies that if target exists, you're done;
conversely, "redo target" in English implies you really
want to *redo* it, not just sit around.
- If this weren't the rule, `redo` and `redo-ifchange`
would mean the same thing, which seems rather confusing.
- If `redo` could refuse to run a .do script, you would
have no easy one-line way to force a particular target to
be rebuilt. You'd have to remove the target and *then*
redo it, which is more typing. On the other hand, nobody
actually types "redo foo.o" if they honestly think foo.o
doesn't need rebuilding.
- For "contentless" targets like "test" or "clean", it would
be extremely confusing if they refused to run just
because they ran successfully last time.
In make, things get complicated because it doesn't
differentiate between these two modes. Makefile rules
with no dependencies run every time, *unless* the target
exists, in which case they run never, *unless* the target
is marked ".PHONY", in which case they run every time. But
targets that *do* have dependencies follow totally
different rules. And all this is needed because there's no
way to tell make, "Listen, I just really want you to run
the rules for this target *right now*."
With redo, the semantics are really simple to explain. If
your brain has already been fried by make, you might be
surprised by it at first, but once you get used to it, it's
really much nicer this way.
# Can I write .do files in my favourite language, not sh?
Yes. If the first line of your .do file starts with the
magic "#!/" sequence (eg. `#!/usr/bin/python`), then redo
will execute your script using that particular interpreter.
Note that this is slightly different from normal Unix
execution semantics. redo never execs your script directly;
it only looks for the "#!/" line. The main reason for this
is so that your .do scripts don't have to be marked
executable (chmod +x). Executable .do scripts would
suggest to users that they should run them directly, and
they shouldn't; .do scripts should always be executed
inside an instance of redo, so that dependencies can be
tracked correctly.
WARNING: If your .do script *is* written in Unix sh, we
recommend *not* including the `#!/bin/sh` line. That's
because there are many variations of /bin/sh, and not all
of them are POSIX compliant. redo tries pretty hard to
find a good default shell that will be "as POSIXy as
possible," and if you override it using #!/bin/sh, you lose
this benefit and you'll have to worry more about
portability.
# Can a single .do script generate multiple outputs?
FIXME: Yes, but this is a bit imperfect.
For example, compiling a .java file produces a bunch of .class
files, but exactly which files? It depends on the content
of the .java file. Ideally, we would like to allow our .do
file to compile the .java file, note which .class files
were generated, and tell redo about it for dependency
checking.
However, this ends up being confusing; if myprog depends
on foo.class, we know that foo.class was generated from
bar.java only *after* bar.java has been compiled. But how
do you know, the first time someone asks to build myprog,
where foo.class is supposed to come from?
So we haven't thought about this enough yet.
Note that it's *okay* for a .do file to produce targets
other than the advertised one; you just have to be careful.
You could have a default.javac.do that runs 'javac
$2.java', and then have your program depend on a bunch of .javac
files. Just be careful not to depend on the .class files
themselves, since redo won't know how to regenerate them.
This feature would also be useful, again, with ./configure:
typically running the configure script produces several
output files, and it would be nice to declare dependencies
on all of them.
# Should I use environment variables to affect my build?
Directly using environment variables is a bad idea because you can't declare
dependencies on them. Also, if there were a file that contained a set of
variables that all your .do scripts need to run, then `redo` would have to
read that file every time it starts (which is frequently, since it's
recursive), and that could get slow.
Luckily, there's an alternative. Once you get used to it, this method is
actually much better than environment variables, because it runs faster
*and* it's easier to debug.
For example, djb often uses a computer-generated script called `compile` for
compiling a .c file into a .o file. To generate the `compile` script, we
create a file called `compile.do`:
redo-ifchange config.sh
. ./config.sh
echo "gcc -c -o \$3 \$2.c $CFLAGS" >$3
chmod a+x $3
Then, your `default.o.do` can simply look like this:
redo-ifchange compile $2.c
./compile $1 $2 $3
This is not only elegant, it's useful too. With make, you have to always
output everything it does to stdout/stderr so you can try to figure out
exactly what it was running; because this gets noisy, some people write
Makefiles that deliberately hide the output and print something friendlier,
like "Compiling hello.c". But then you have to guess what the compile
command looked like.
With redo, the command *is* `./compile hello.c`, which looks good when
printed, but is also completely meaningful. Because it doesn't depend on
any environment variables, you can just run `./compile hello.c` to reproduce
its output, or you can look inside the `compile` file to see exactly what
command line is being used.
As a bonus, all the variable expansions only need to be done once: when
generating the ./compile program. With make, it would be recalculating
expansions every time it compiles a file. Because of the
way make does expansions as macros instead of as normal
variables, this can be slow.
# Example default.o.do for both C and C++ source?
We can upgrade the compile.do from the previous answer to
look something like this:
redo-ifchange config.sh
. ./config.sh
cat <<-EOF
[ -e "\$2.cc" ] && EXT=.cc || EXT=.c
gcc -o "\$3" -c "\$1\$EXT" -Wall $CFLAGS
EOF
chmod a+x "$3"
Isn't it expensive to have ./compile doing this kind of test for every
single source file? Not really. Remember, if you have two implicit rules
in make:
%.o: %.cc
gcc ...
%.o: %.c
gcc ...
Then it has to do all the same checks. Except make has even *more* implicit
rules than that, so it ends up trying and discarding lots of possibilities
before it actually builds your program. Is there a %.s? A
%.cpp? A %.pas? It needs to look for *all* of them, and
it gets slow. The more implicit rules you have, the slower
make gets.
In redo, it's not implicit at all; you're specifying exactly how to
decide whether it's a C program or a C++ program, and what to do in each
case. Plus you can share the two gcc command lines between the two rules,
which is hard in make. (In GNU make you can use macro functions, but the
syntax for those is ugly.)
# Can I just rebuild just part of a project?
Absolutely! Although `redo` runs "top down" in the sense of one .do file
calling into all its dependencies, you can start at any point in the
dependency tree that you want.
Unlike recursive make, no matter which subdir of your project you're in when
you start, `redo` will be able to build all the dependencies in the right
order.
Unlike non-recursive make, you don't have to jump through any strange hoops
(like adding, in each directory, a fake Makefile that does `make -C ${TOPDIR}`
back up to the main non-recursive Makefile). redo just uses `filename.do`
to build `filename`, or uses `default*.do` if the specific `filename.do`
doesn't exist.
When running any .do file, `redo` makes sure its current directory is set to
the directory where the .do file is located. That means you can do this:
redo ../utils/foo.o
And it will work exactly like this:
cd ../utils
redo foo.o
In make, if you run
make ../utils/foo.o
it means to look in ./Makefile for a rule called
../utils/foo.o... and it probably doesn't have such a
rule. On the other hand, if you run
cd ../utils
make foo.o
it means to look in ../utils/Makefile and look for a rule
called foo.o. And that might do something totally
different! redo combines these two forms and does
the right thing in both cases.
Note: redo will always change to the directory containing
the .do file before trying to build it. So if you do
redo ../utils/foo.o
the ../utils/default.o.do file will be run with its current directory set to
../utils. Thus, the .do file's runtime environment is
always reliable.
On the other hand, if you had a file called ../default.o.do,
but there was no ../utils/default.o.do, redo would select
../default.o.do as the best matching .do file. It would
then run with its current directory set to .., and tell
default.o.do to create an output file called "utils/foo.o"
(that is, foo.o, with a relative path explaining how to
find foo.o when you're starting from the directory
containing the .do file).
That sounds a lot more complicated than it is. The results
are actually very simple: if you have a toplevel
default.o.do, then all your .o files will be compiled with
$PWD set to the top level, and all the .o filenames passed
as relative paths from $PWD. That way, if you use relative
paths in -I and -L gcc options (for example), they will
always be correct no matter where in the hierarchy your
source files are.
# Can I put my .o files in a different directory from my .c files?
Yes. There's nothing in redo that assumes anything about
the location of your source files. You can do all sorts of
interesting tricks, limited only by your imagination. For
example, imagine that you have a toplevel default.o.do that looks
like this:
ARCH=${1#out/}
ARCH=${ARCH%%/*}
SRC=${1#out/$ARCH/}
redo-ifchange $SRC.c
$ARCH-gcc -o $3 -c $SRC.c
If you run `redo out/i586-mingw32msvc/path/to/foo.o`, then
the above script would end up running
i586-mingw32msvc-gcc -o $3 -c path/to/foo.c
You could also choose to read the compiler name or options from
out/$ARCH/config.sh, or config.$ARCH.sh, or use any other
arrangement you want.
You could use the same technique to have separate build
directories for out/debug, out/optimized, out/profiled, and so on.
# Can my filenames have spaces in them?
Yes, unlike with make. For historical reasons, the Makefile syntax doesn't
support filenames with spaces; spaces are used to separate one filename from
the next, and there's no way to escape these spaces.
Since redo just uses sh, which has working escape characters and
quoting, it doesn't have this problem.
# Does redo care about the differences between tabs and spaces?
No.
# What if my .c file depends on a generated .h file?
This problem arises as follows. foo.c includes config.h, and config.h is
created by running ./configure. The second part is easy; just write a
config.h.do that depends on the existence of configure (which is created by
configure.do, which probably runs autoconf).
The first part, however, is not so easy. Normally, the headers that a C
file depends on are detected as part of the compilation process. That works
fine if the headers, themselves, don't need to be generated first. But if
you do
redo foo.o
There's no way for redo to *automatically* know that compiling foo.c
into foo.o depends on first generating config.h.
Since most .h files are *not* auto-generated, the easiest
thing to do is probably to just add a line like this to
your default.o.do:
redo-ifchange config.h
Sometimes a specific solution is much easier than a general
one.
If you really want to solve the general case,
[djb has a solution for his own
projects](http://cr.yp.to/redo/honest-nonfile.html), which is a simple
script that looks through C files to pull out #include lines. He assumes
that `#include <file.h>` is a system header (thus not subject to being
built) and `#include "file.h"` is in the current directory (thus easy to
find). Unfortunately this isn't really a complete
solution, but at least it would be able to redo-ifchange a
required header before compiling a program that requires
that header.

52
docs/GettingStarted.md Normal file
View file

@ -0,0 +1,52 @@
# Prerequisites
Currently, this version of redo requires python2.7 and the python2.7 sqlite3 module.
Optional, but recommended, is the
[setproctitle](http://code.google.com/p/py-setproctitle/) module, which makes your
`ps` output prettier.
In modern versions of Debian, sqlite3 is already part of the python2.7 package.
You can install the prerequisites like this:
```sh
sudo apt-get install python2.7 python-setproctitle
```
(If you have install instructions for other OSes, please add them here :))
# Clone, compile, and test redo
You can run redo without installing it, like this:
```sh
git clone https://github.com/apenwarr/redo
cd redo
./redo -j10 test
```
If the tests pass, you can either add $PWD/redo to your PATH, or install
redo on your system. To install for all users, put it in /usr/local:
```sh
PREFIX=/usr/local sudo ./redo install
```
Or to install it just for yourself (without needing root access), put it in
your home directory:
```sh
PREFIX=$HOME ./redo install
```
# Pre-compiled packages
## MacOS
redo is available from the [Homebrew](https://brew.sh/) project:
brew install redo
## Linux
Various linux distributions include redo under different names. Most of the
packages are unfortunately obsolete and don't contain the most recent bug
fixes. At this time (late 2018), we recommend using the latest tagged
version [from github](https://github.com/apenwarr/redo).

329
docs/Roadmap.md Normal file
View file

@ -0,0 +1,329 @@
We believe that, unlike most programs, it's actually possible to "finish"
redo, in the sense of eventually not needing to extend its semantics or add
new features. That's because redo is a pretty low-level system that just
provides some specific features (dependency checking, parallelism, log
linearization, inter-process locking). It's the job of your build scripts
to tie those features together in the way you want.
`make` has its own imperative syntax, which creates a temptation to add
new built-in functions and syntax extensions. In more
"declarative" build systems, there's a constant need to write new extension
modules or features in order to create functionality that wasn't available
declaratively. redo avoids that by using a turing-complete language
to run your builds. You should be able to build anything at all with redo,
just by writing your .do scripts the way you want.
Thus, the only things that need to be added to redo (other than portability
and bug fixes, which will likely be needed forever) are to fix gaps in
redo's model that prevent you from getting your work done. This document
describes the gaps we're currently aware of.
Note that all of the items in this document are still unimplemented. In
most cases, that's because we haven't yet settled on a final design, and
it's still open to discussion. The place to discuss design issues is the
[mailing list](../Contributing/#mailing-list).
### default.do search path, and separated build directories
One of the most controversial topics in redo development, and for developers
trying to use redo, is: where do you put all those .do scripts?
redo searches hierarchically up the directory tree from a target's filename,
hoping to find a `default*.do` file that will match, and then uses the first
one it finds. This method is rather elegant when it works. But many
developers would like to put their output files into a separate directory
from their source files, and that output directory might not be a
subdirectory of the main project (for example, if the main project is on a
read-only filesystem).
There are already a few ways to make this work, such as placing a single
`default.do` "proxy" or "delegation" script at the root of the output
directory, which will bounce requests to .do files it finds elsewhere. One
nice thing about this feature is it doesn't require any changes to redo
itself; redo already knows how to call your toplevel default.do script.
However, some people find the delegation script to be inelegant and
complicated.
Other options include searching inside a known subdirectory name (eg.
`do/`), which could be a symlink; or adding a `.dopath` file which tells
redo to search elsewhere.
So far, we haven't settled on the best design, and discussion is welcome.
In the meantime, you can write a delegation script (TODO: link to example)
for your project. Because this requires no special redo features, it's
unlikely to break in some later version of redo, even if we add a new
method.
### .do files that produce directories
Sometimes you want a .do file to produce multiple output files in a single
step. One example is an autoconf `./configure` script, which might produce
multiple files. Or, for example, look at the [LaTeX typesetting
example](../cookbook/latex/) in the redo cookbook.
In the purest case, generating multiple outputs from a single .do file
execution violates the redo semantics. The design of redo calls for
generating *one* output file from *zero or more* input files. And most of
the time, that works fine. But sometimes it's not enough.
Currently (like in the LaTeX example linked above) we need to resolve this
problem by taking advantage of "side effects" in redo: creating a set of
files that are unknown to redo, but sit alongside the "known" files in the
filesystem. But this has the annoying limitation that you cannot
redo-ifchange directly on the file you want, if it was generated this way.
For example, if `runconfig.do` generates `Makefile` and `config.h`, you
must not `redo-ifchange config.h` directly; there is no .do file for
`config.h`. You must `redo-ifchange runconfig` and then *use*
`config.h`.
(There are workarounds for that workaround: for example, `runconfig.do`
could put all its output files in a `config/` directory, and then you could
have a `config.h.do` that does `redo-ifchange runconfig` and `cp
config/config.h $3`. Then other scripts can `redo-ifchange config.h`
without knowing any more about it. But this method gets tedious.)
One suggestion for improving the situation would be to teach redo about
"directory" targets. For example, maybe we have a `config.dir.do` that
runs `./configure` and produces files in a directory named `config`. The
`.dir.do` magic suffix tells redo that if someone asks for
`config/config.h`, it must first try to instantiate the directory named
`config` (using `config.dir.do`), and only then try to depend on the file
inside that directory.
There are a number of holes in this design, however. Notably, it's not
obvious how redo should detect when to activate the magic directory feature.
It's easy when there is a file named `config.dir.do`, but much less obvious
for a file like `default.dir.do` that can construct certain directory types,
but it's not advertised which ones.
This particular cure may turn out to be worse than the disease.
### Per-target-directory .redo database
An unexpectedly very useful feature of redo is the ability to "redo from
anywhere" and get the same results:
```shell
$ cd /a/b/c
$ redo /x/y/z/all
```
should have the same results as
```shell
$ cd /x/y/z
$ redo all
```
Inside a single project, this already works. But as redo gets used more
widely, and in particular when you have multiple redo-using projects that
want to refer to other redo-using projects, redo can get confused about
where to put its `.redo` state database. Normally, it goes into a directory
called `$REDO_BASE`, the root directory of your project. But if a .do
script refers to a directory outside or beside the root, this method doesn't
work, and redo gets the wrong file state information.
Further complications arise in the case of symlinks. For example, if you
ask redo to build `x/y/z/file` but `y` is a symlink to `q`, then redo will
effectively end up replacing `x/q/z/file` when it replces `x/y/z/file`,
since they're the same. If someone then does `redo-ifchange x/q/z/file`,
redo may become confused about why that file has "unexpectedly" changed.
The fix for both problems is simple: put one `.redo` database in every
directory that contains target files. The `.redo` in each directory
contains information only about the targets in that directory. As a result,
`x/y/z/file` and `x/q/z/file` will share the same state database,
`x/q/z/.redo`, and building either target will update the state database's
understanding of the file called `file` in the same directory, and there
will be no confusion.
Similarly, one redo-using project can refer to targets in another redo-using
project with no problem, because redo will no longer have the concept of a
`$REDO_BASE`, so there is no way to talk about targets "outside" the
`$REDO_BASE`.
Note that there is no reason to maintain a `.redo` state database in
*source* directories (which might be read-only), only target directories.
This is because we store `stat(2)` information for each dependency anyway, so
it's harmless if multiple source filenames are aliases for the same
underlying content.
### redo-{sources,targets,ood} should take a list of targets
With the above change to a per-target-directory `.redo` database, the
original concept of the `redo-sources`, `redo-targets`, and `redo-ood`
commands needs to change. Currently they're defined to list "all" the
sources, targets, and out-of-date targets, respectively. But when there is
no single database reflecting the entire filesystem, the concept of "all"
becomes fuzzy.
We'll have to change these programs to refer to "all (recursive)
dependencies of targets in the current directory" by default, or of all
targets listed on the command line otherwise. This is probably more useful
than the current behaviour anyway, since in a large project, one rarely
wants to see a complete list of all sources and targets.
### Deprecating "stdout capture" behaviour
The [original design for redo](http://cr.yp.to/redo.html) specified that a
.do script could produce its output either by writing to stdout, or by
writing to the file named by the `$3` variable.
Experience has shown that most developers find this very confusing. In
particular, results are undefined if you write to *both* stdout and `$3`.
Also, many programs (including `make`!) write their log messages to stdout
when they should write to stderr, so many .do scripts need to start with
`exec >&2` to avoid confusion.
In retrospect, automatically capturing stdout was probably a bad idea. .do
scripts should intentionally redirect to `$3`. To enforce this, we could
have redo report an error whenever a .do script returns after writing to its
stdout. For backward compatibility, we could provide a command-line option
to downgrade the error to a warning.
### Deprecating environment variable sharing
In redo, it's considered a poor practice to pass environment variables (and
other process attributes, like namespaces) from one .do script to another.
This is because running `redo-ifchange /path/to/file` should always run
`file`'s .do script with exactly the same settings, whether you do it from
the toplevel from from deep inside a tree of dependencies. If an
environment variable set in one .do script can change what's seen by an
inner .do script, this breaks the dependency mechanism and makes builds less
repeatable.
To make it harder to do this by accident, redo could intentionally wipe all
but a short whitelist of allowed environment variables before running any
.do script.
As a bonus, by never sharing any state outside the filesystem, it becomes
much more possible to make a "distributed redo" that builds different
targets on different physical computers.
### redo-recheck command
Normally, redo only checks any given file dependency at most once per
session, in order to reduce the number of system calls executed, thus
greatly speeding up incremental builds. As a result, `redo-ifchange` of the
same target will only execute the relevant .do script at most once per
session.
In some situations, notably integration tests, we want to force redo to
re-check more often. Right now there's a hacky script called
`t/flush-cache` in the redo distribution which does this, but it relies on
specific knowledge of the .redo directory's database format, which means it
only works in this specific version of redo; this prevents the integration
tests from running (and thus checking compatibility with) competing redo
implementations.
If we standardized a `redo-recheck` command, which would flush the cache for
the targets given on the command line, and all of their dependencies, this
sort of integration test could work across multiple redo versions. For redo
versions which don't bother caching, `redo-recheck` could be a null
operation.
### tty input
Right now, redo only allows a .do file to request input from the user's
terminal if using `--no-log` and *not* using the `-j` option. Terminal
input is occasionally useful for `make config` interfaces, but parallelism
and log linearization make the console too cluttered for a UI to work.
The ninja build system has a [console
pool](https://ninja-build.org/manual.html#_the_literal_console_literal_pool)
that can contain up to one job at a time. When a job is in the console
pool, it takes over the console entirely.
We could probably implement something similar in redo by using POSIX job
control features, which suspend subtasks whenever they try to read
from the tty. If we caught the suspension signal and acquired a lock, we
could serialize console access.
Whether the complexity of this feature is worthwhile is unclear. Maybe it
makes more sense just to have a './configure' script that runs outside the
redo environment, but still can call into redo-ifchange if needed.
### redo-lock command
Because it supports parallelism via recursion, redo automatically handles
inter-process locking so that only one instance of redo can try to build a
given target at a time.
This sort of locking turns out to be very useful, but there are a few
situations where requiring redo to "build a target by calling a .do file" in
order to acquire a lock becomes awkward.
For example, imagine redo is being used to call into `make` to run arbitrary
`Makefile` targets. `default.make.do` might look like this:
```sh
make "$2"
```
redo will automatically prevent two copies of `redo all.make` from running
at once. However, if someone runs `redo all.make myprogram.make`, then two
copies of `make` will execute at once. This *might* be harmless, but if the
`all` target in the `Makefile` has a dependency on `myprogram`, then we will
actually end up implicitly building `myprogram` from two places at once:
from the `myprogram` part of `all.make` and from `myprogram.make`.
In hairy situations like that, it would be nice to serialize all access
inside `default.make.do`, perhaps like this:
```sh
redo-lock make.lock make "$2"
```
This would create a redo-style lock on the (virtual) file `make.lock`, but
then instead of trying to `redo make.lock`, it would run the given command,
in this case `make "$2"`.
It's unclear whether this feature is really a good idea. There are other
(convoluted) ways to achieve the same goal. Nevertheless, it would be easy
enough to implement. And redo versions that don't support parallelism could
just make redo-lock a no-op, since they guarantee serialization in all cases
anyway.
### Include a (minimal) POSIX shell
A common problem when writing build scripts, both in `make` and in redo, is
gratuitous incompatibility between all the available POSIX-like unix shells.
Nowadays, most shells support [various pure POSIX sh
features](https://apenwarr.ca/log/20110228), but there are always glitches.
In some cases, POSIX doesn't define the expected behaviour for certain
situations. In others, shells like `bash` try to "improve" things by
changing the syntax in non-POSIX ways. Or maybe they just add new
backward-compatible features, which you then rely on accidentally because
you only tested your scripts with `bash`.
redo on Windows using something like [MSYS](http://www.mingw.org/wiki/msys)
is especially limited by the lack of (and oddity of) available unix tools.
To avoid all these portability problems for .do script maintainers, we might
consider bundling redo with a particular (optional) sh implementation, and
maybe also unix-like tools, that it will use by default. An obvious
candidate would be busybox, which has a win32 version called
[busybox-w32](https://frippery.org/busybox/).
### redoconf
redo is fundamentally a low-level tool that doesn't know as much about
compiling specific programming languages as do higher-level tools like
[cmake](https://cmake.org/).
Similarly, `make` doesn't know much about specific programming languages
(and what it does know is hopelessly out of date, but cannot be deleted or
updated because it would break backward compatibility with old Makefiles).
This is why `autoconf` and `automake` were created: to automatically fill in
the language- and platform-specific blanks, while letting `make` still
handle executing the low level instructions.
It might be useful to have a redo-native autoconf/automake-like system,
although you can already use autoconf with redo, so this might not be
essential.

9
docs/all.do Normal file
View file

@ -0,0 +1,9 @@
redo-ifchange doc.list
sed 's/\.md/.1/' <doc.list |
xargs redo-ifchange
# mkdocs foolishly tries to process every file in this directory, which
# leads it to try to open temp files produced by the above redo-ifchange
# if it runs in parallel with those jobs. So don't run it until they
# finish.
redo-ifchange mkdocs

2
docs/clean.do Normal file
View file

@ -0,0 +1,2 @@
redo cookbook/clean
rm -f *~ .*~ *.1 t/*.1 md-to-man *.tmp t/*.tmp *.html */*.html *.list

3
docs/cookbook/all.do Normal file
View file

@ -0,0 +1,3 @@
for d in */all.do; do
echo "${d%.do}"
done | xargs redo-ifchange

3
docs/cookbook/clean.do Normal file
View file

@ -0,0 +1,3 @@
for d in */clean.do; do
echo "${d%.do}"
done | xargs redo

5
docs/cookbook/defaults/.gitignore vendored Normal file
View file

@ -0,0 +1,5 @@
date
version
include/version.h
version.py
test.txt

2
docs/cookbook/defaults/all.do Normal file
View file

@ -0,0 +1,2 @@
redo-ifchange test.txt version.py \
test.py include/version.h

2
docs/cookbook/defaults/clean.do Normal file
View file

@ -0,0 +1,2 @@
rm -f date version include/version.h \
version.py test.txt *~ .*~

3
docs/cookbook/defaults/date.do Normal file
View file

@ -0,0 +1,3 @@
date +'%Y-%m-%d' >$3
redo-always
redo-stamp <$3

25
docs/cookbook/defaults/default.do Normal file
View file

@ -0,0 +1,25 @@
# $1 is the target name, eg. test.txt
# $2 in the same as $1. We'll talk about
# that in a later example.
# $3 is the temporary output file we should
# create. If this script is successful,
# redo will atomically replace $1 with $3.
if [ -e "$1.in" ]; then
# if a .in file exists, then do some
# text substitution.
#
# Remember, the user asks redo to build
# a particular *target* name. It's the .do
# file's job to figure out what source file(s)
# to use to generate the target.
redo-ifchange "$1.in" version date
read VERSION <version
read DATE <date
sed -e "s/%VERSION%/$VERSION/g" \
-e "s/%DATE%/$DATE/g" \
<$1.in >$3
else
echo "$0: Fatal: don't know how to build '$1'" >&2
exit 99
fi

8
docs/cookbook/defaults/include/version.h.in Normal file
View file

@ -0,0 +1,8 @@
// C/C++ header file identifying the current version
#ifndef __VERSION_H
#define __VERSION_H
#define VERSION "%VERSION%"
#define DATE "%DATE%"
#endif // __VERSION_H

322
docs/cookbook/defaults/index.md Normal file
View file

@ -0,0 +1,322 @@
### Building a text preprocessor
Let's say we want to process a bunch of files, each in the same way. An
easy example is a text preprocessor: we'll look through the text and replace
variables of the form %VARNAME% with real content. In our example, we'll
use variable %VERSION%, representing a version number, and %DATE%, with
today's date.
To play with this code on your own machine, get the [redo
source code](https://github.com/apenwarr/redo) and look in the
`docs/cookbook/defaults/` directory.
### Input files
Let's create some input files we can use. The input to our
preprocessor will have file extension `.in`. The output can be in various
formats.
<pre><code src='test.txt.in'></code></pre>
<pre><code lang='c' src='include/version.h.in'></code></pre>
<pre><code lang='python' src='version.py.in'></code></pre>
For fun, let's also make a python test program that calls the newly
generated version.py:
<pre><code lang='python' src='test.py'></code></pre>
Finally, we need to provide values for the variables. Let's put each one
in its own file, named `version` and `date`, respectively.
<pre><b style='text-align: center; display: block'>version</b
><code>1.0</code></pre>
<pre><b style='text-align: center; display: block'>date</b
><code>1970-01-01</code></pre>
### default.do files
Now we want to teach redo how to do our substitutions: how to, in general,
generate file X from file X.in.
We could write a separate .do file for every output file X. For example, we
might make `test.txt.do` and `version.py.do`. But that gets tedious. To
make it easier, if there is no specific `X.do` for a target named X, redo
will try using `default.do` instead. Let's write a default.do for our text
preprocessor.
<pre><code lang='sh' src='default.do'></code></pre>
If `default.do` is asked to build `X`, and there exists a file named `X.in`, we
use `sed` to do our variable substitutions. In that case, `default.do` uses
redo-ifchange to depend on `X.in`, `version`, and `date`. If a file named
`X.in` does *not* exist, then we don't know what to do, so we give an error.
On the other hand, if we try to generate a file that *already* exists, like
`test.py`, redo does not call default.do at all. redo only tries to create
files that don't exist, or that were previously generated by redo. This
stops redo from accidentally overwriting your work.
```shell
$ redo test.txt
redo test.txt
$ cat test.txt
This is the documentation for MyProgram version
1.0. It was generated on 1970-01-01.
$ redo chicken
redo chicken
default.do: Fatal: don't know how to build 'chicken'
redo chicken (exit 99)
$ redo version.py
redo version.py
# test.py was created by us, so it's a "source" file.
# redo does *not* call default.do to replace it.
$ redo test.py
redo: test.py: exists and not marked as generated; not redoing.
$ python test.py
Version '1.0' has build date '1970-01-01'
```
Nice!
While we're here, let's make an `all.do` so that we don't have to tell redo
exactly which files to rebuild, every single time.
<pre><code lang='sh' src='all.do'></code></pre>
Results:
```shell
$ redo
redo all
redo test.txt
redo version.py
redo include/version.h
# input files didn't change, so nothing to rebuild
$ redo
redo all
$ touch test.txt.in
$ redo
redo all
redo test.txt
```
### Auto-generating the version and date (redo-always and redo-stamp)
Of course, in a real project, we won't want to hardcode the version number
and date into a file. Ideally, we can get the version number from a version
control system, like git, and we can use today's date.
To make that happen, we can replace the static `version` and `date` files with
`version.do` and `date.do`. `default.do` already uses redo-ifchange to
depend on `version` and `date`, so redo will create them as needed, and
if they change, redo will rebuild all the targets that depend on them.
However, the `version` and `date` files are special: they depend on the
environment outside redo itself. That is, there's no way to declare a
dependency on the current date. We might generate the `date` file once, but
tomorrow, there's no way for redo to know that its value should change.
To handle this situation, redo has the `redo-always` command. If we run
redo-always from a .do file, it means every time someone depends on that
target, it will be considered out-of-date and need to be rebuilt. The
result looks like this:
```shell
$ redo
redo all
redo test.txt
redo version
redo date
redo version.py
redo include/version.h
# version.do and date.do are redo-always, so
# everything depending on them needs to rebuild
# every time.
$ redo
redo all
redo test.txt
redo version
redo date
redo version.py
redo include/version.h
```
Of course, for many uses, that's overcompensating: the version number and
date don't change *that* often, so we might end up doing a lot of
unnecessary work on every build. To solve that, there's `redo-stamp`.
redo-stamp does the opposite of redo-always: while redo-always makes things
build *more* often, redo-stamp makes things build *less* often.
Specifically, it lets a .do file provide a "stamp value" for its output; if
that stamp value is the same as before, then the target should be considered
unchanged after all.
The most common stamp value is just the content itself. Since in redo, we
write the content to $3, we can also read it back from $3:
<pre><code lang='sh' src='version.do'></code></pre>
<pre><code lang='sh' src='date.do'></code></pre>
And the final result is what we want. Although `version` and `date` are
generated every time, the targets which depend on them are not:
```shell
$ redo clean
redo clean
# version and date are generated just once per run,
# the first time they are used.
$ redo
redo all
redo test.txt
redo version
redo date
redo version.py
redo include/version.h
# Here, (test.txt) means redo is considering building
# test.txt, but can't decide yet. In order to decide,
# it needs to first build date and version. After
# that, it decides not to build test.txt after all.
$ redo
redo all
redo (test.txt)
redo date
redo version
```
### Temporary overrides
Sometimes you want to override a file even if it *is* a target (ie. it has
previously been built by redo and has a valid .do file associated with it).
In our example, maybe you want to hardcode the version number because you're
building a release. This is easy: redo notices whenever you overwrite a
file from outside redo, and will avoid replacing that file until you
subsequently delete it:
```shell
$ echo "1.0" >version
$ redo
redo all
redo (test.txt)
redo date
redo: version - you modified it; skipping
redo test.txt
redo version.py
redo include/version.h
$ redo
redo all
redo (test.txt)
redo date
redo: version - you modified it; skipping
$ rm version
$ redo
redo all
redo (test.txt)
redo date
redo version
redo test.txt
redo version.py
redo include/version.h
```
### default.do, subdirectories, and redo-whichdo
There's one more thing we should mention, which is the interaction of
default.do with files in subdirectories. Notice that we are building
`include/version.h` in our example:
```shell
$ redo include/version.h
redo include/version.h
redo version
redo date
$ cat include/version.h
// C/C++ header file identifying the current version
#ifndef __VERSION_H
#define __VERSION_H
#define VERSION "redo-0.31-3-g974eb9f"
#define DATE "2018-11-26"
#endif // __VERSION_H
```
redo works differently from the `make` command when you ask it to build
files in subdirectories. In make's case, it always looks for a `Makefile`
in the *current* directory, and uses that for all build instructions. So
`make include/version.h` and `cd include && make version.h` are two
different things; the first uses `Makefile`, and the second uses
`include/Makefile` (or crashes if the latter does not exist).
redo, on the other hand, always uses the same formula to find a .do file for
a particular target. For a file named X, that formula is as follows:
- first, try X.do
- then try default.do
- then try ../default.do
- then try ../../default.do
- ...and so on...
(Note: for targets with an extension, like X.o, redo actually tries even
more .do files, like `default.o.do` and `../default.o.do`. For precise
details, read the [redo man page](../../redo).)
You can see which .do files redo considers for a given target by using the
`redo-whichdo` command. If redo-whichdo returns successfully, the last name
in the list is the .do file it finally decided to use.
```shell
$ redo-whichdo include/version.h
include/version.h.do
include/default.h.do
include/default.do
default.h.do
default.do
```
### Redo always runs in the .do file's directory
To ensure consistency, redo always changes the current directory to
the directory *containing the selected .do file* (**not** the directory
containing the target, if they are different). As a result,
`redo include/version.h` and `cd include && redo version.h` always have
exactly the same effect:
```shell
$ redo include/version.h
redo include/version.h
redo version
redo date
$ (cd include && redo version.h)
redo version.h
redo ../version
redo ../date
```
(redo's display is slightly different between the two: it always shows the
files it's building relative to the $PWD at the time you started redo.)
This feature is critical to redo's recursive nature; it's the reason that
essays like [Recursive Make Considered Harmful](http://aegis.sourceforge.net/auug97.pdf)
don't apply to redo. Any redo target, anywhere in your source tree, can
use redo-ifchange to depend on any of your other targets, and the dependency
will work right.
Why does redo change to the directory containing the .do file, instead of
the directory containing the target? Because usually, the .do file needs to
refer to other dependencies, and it's easier to always express those
dependencies without adjusting any paths. In our text preprocessor example,
`default.do` does `redo-ifchange version date`; this wouldn't work properly
if it were running from the `include/` directory, because there are no files
named `version` and `date` in there.
Similarly, when compiling C programs, there are
usually compiler options like `-I../mylib/include`. If we're compiling
`foo.o` and `mydir/bar.o`, we would like `-I../mylib/include` to have the
same meaning in both cases.

6
docs/cookbook/defaults/test.py Normal file
View file

@ -0,0 +1,6 @@
#!/usr/bin/env python
"""Test program for auto-generated version.py"""
import version
print('Version %r has build date %r'
% (version.VERSION, version.DATE))

2
docs/cookbook/defaults/test.txt.in Normal file
View file

@ -0,0 +1,2 @@
This is the documentation for MyProgram version
%VERSION%. It was generated on %DATE%.

7
docs/cookbook/defaults/version.do Normal file
View file

@ -0,0 +1,7 @@
# Try to get a version number from git, if possible.
if ! git describe >$3; then
echo "$0: Falling back to static version." >&2
echo 'UNKNOWN' >$3
fi
redo-always
redo-stamp <$3

3
docs/cookbook/defaults/version.py.in Normal file
View file

@ -0,0 +1,3 @@
# python module identifying the current version
VERSION='%VERSION%'
DATE='%DATE%'

3
docs/cookbook/hello/.gitignore vendored Normal file
View file

@ -0,0 +1,3 @@
hello
*~
.*~

1
docs/cookbook/hello/all.do Normal file
View file

@ -0,0 +1 @@
redo-ifchange hello

1
docs/cookbook/hello/clean.do Normal file
View file

@ -0,0 +1 @@
rm -f hello *~ .*~

6
docs/cookbook/hello/hello.c Normal file
View file

@ -0,0 +1,6 @@
#include <stdio.h>
int main() {
printf("Hello, world!\n");
return 0;
}

16
docs/cookbook/hello/hello.do Normal file
View file

@ -0,0 +1,16 @@
# If hello.c changes, this script needs to be
# re-run.
redo-ifchange hello.c
# Compile hello.c into the 'hello' binary.
#
# $3 is the redo variable that represents the
# output filename. We want to build a file
# called "hello", but if we write that directly,
# then an interruption could result in a
# partially-written file. Instead, write it to
# $3, and redo will move our output into its
# final location, only if this script completes
# successfully.
#
cc -o $3 hello.c -Wall

142
docs/cookbook/hello/index.md Normal file
View file

@ -0,0 +1,142 @@
### Hello!
Let's start with Hello World: famously, the simplest project that does
anything interesting. We'll write this one in C, but don't worry if
you're not a C programmer. The focus isn't the C code itself, just to
compile it.
To play with the code on your own machine, get the [redo
source code](https://github.com/apenwarr/redo) and look in the
`docs/cookbook/hello/` directory.
### Compiling the code
First, let's create a source file that we want to compile:
<pre><code lang='c' src='hello.c'></code></pre>
Now we need a .do file to tell redo how to compile it:
<pre><code lang='sh' src='hello.do'></code></pre>
With those files in place, we can build and run the program:
```shell
$ redo hello
redo hello
$ ./hello
Hello, world!
```
Use the `redo` command to forcibly re-run a specific rule (in this case, the
compiler). Or, if you only want to recompile `hello` when its input
files (dependencies) have changed, use `redo-ifchange`.
```shell
$ redo hello
redo hello
# Rebuilds, whether we need it or not
$ redo hello
redo hello
# Does not rebuild because hello.c is unchanged
$ redo-ifchange hello
$ touch hello.c
# Notices the change to hello.c
$ redo-ifchange hello
redo hello
```
Usually we'll want to also provide an `all.do` file. `all` is the
default redo target when you don't specify one.
<pre><code lang='sh' src='all.do'></code></pre>
With that, now we can rebuild our project by just typing `redo`:
```shell
$ rm hello
# 'redo' runs all.do, which calls into hello.do.
$ redo
redo all
redo hello
# Notice that this forcibly re-runs the 'all'
# rule, but all.do calls redo-ifchange, so
# hello itself is only recompiled if its
# dependencies change.
$ redo
redo all
$ ./hello
Hello, world!
```
### Debugging your .do scripts
If you want to see exactly which commands are being run for each step,
you can use redo's `-x` and `-v` options, which work similarly to
`sh -x` and `sh -v`.
```shell
$ rm hello
$ redo -x
redo all
* sh -ex all.do all all all.redo2.tmp
+ redo-ifchange hello
redo hello
* sh -ex hello.do hello hello hello.redo2.tmp
+ redo-ifchange hello.c
+ cc -o hello.redo2.tmp hello.c -Wall
redo hello (done)
redo all (done)
```
### Running integration tests
What about tests? We can, of course, compile a C program that has some
unit tests. But since our program isn't very complicated, let's write
a shell "integration test" (also known as a "black box" test) to make
sure it works as expected, without depending on implementation details:
<pre><code lang='sh' src='test.do'></code></pre>
Even if we rewrote our hello world program in python, javascript, or
ruby, that integration test would still be useful.
### Housekeeping
Traditionally, it's considered polite to include a `clean` rule that
restores your project to pristine status, so people can rebuild from
scratch:
<pre><code lang='sh' src='clean.do'></code></pre>
Some people like to include a `.gitignore` file so that git won't pester
you about files that would be cleaned up by `clean.do` anyway. Let's add
one:
<pre><b align=center style="display: block">.gitignore</b><code>
hello
*~
.*~
</code></pre>
Congratulations! That's all it takes to make your first redo project.
Here's what it looks like when we're done:
```shell
$ ls
all.do clean.do hello.c hello.do test.do
```
Some people think this looks a little cluttered with .do files. But
notice one very useful feature: you can see, at a glance, exactly which
operations are possible in your project. You can redo all, clean,
hello, or test. Since most people downloading your project will just
want to build it, it's helpful to have the available actions so
prominently displayed. And if they have a problem with one of the
steps, it's very obvious which file contains the script that's causing
the problem.

12
docs/cookbook/hello/test.do Normal file
View file

@ -0,0 +1,12 @@
# Make sure everything has been built before we start
redo-ifchange all
# Ensure that the hello program, when run, says
# hello like we expect.
if ./hello | grep -i 'hello' >/dev/null; then
echo "success" >&2
exit 0
else
echo "missing 'hello' message!" >&2
exit 1
fi

7
docs/cookbook/latex/.gitignore vendored Normal file
View file

@ -0,0 +1,7 @@
*.eps
*.dvi
*.ps
*.pdf
*.tmp
*~
.*~

8
docs/cookbook/latex/all.do Normal file
View file

@ -0,0 +1,8 @@
for d in latex dvips dvipdf Rscript; do
if ! type "$d" >/dev/null 2>/dev/null; then
echo "$0: skipping: $d not installed." >&2
exit 0
fi
done
redo-ifchange paper.pdf paper.ps

2
docs/cookbook/latex/clean.do Normal file
View file

@ -0,0 +1,2 @@
rm -f *.eps *.dvi *.ps *.pdf *~ .*~
rm -rf *.tmp

2
docs/cookbook/latex/default.dvi.do Normal file
View file

@ -0,0 +1,2 @@
redo-ifchange "$2.runtex"
ln "$2.tmp/$2.dvi" "$3"

3
docs/cookbook/latex/default.pdf.do Normal file
View file

@ -0,0 +1,3 @@
exec >&2
redo-ifchange "$2.dvi"
dvipdf "$2.dvi" "$3"

3
docs/cookbook/latex/default.ps.do Normal file
View file

@ -0,0 +1,3 @@
exec >&2
redo-ifchange "$2.dvi"
dvips -o "$3" "$2.dvi"

58
docs/cookbook/latex/default.runtex.do Normal file
View file

@ -0,0 +1,58 @@
# latex produces log output on stdout, which is
# not really correct. Send it to stderr instead.
exec >&2
# We depend on both the .latex file and its .deps
# file (which lists additional dependencies)
redo-ifchange "$2.latex" "$2.deps"
# Next, we have to depend on each dependency in
# the .deps file.
cat "$2.deps" | xargs redo-ifchange
tmp="$2.tmp"
rm -rf "$tmp"
mkdir -p "$tmp"
# latex generates eg. the table of contents by
# using a list of references ($2.aux) generated
# during its run. The first time, the table of
# contents is empty, so we have to run again.
# But then the table of contents is non-empty,
# which might cause page numbers to change, and
# so on. So we have to keep re-running until it
# finally stops changing.
touch "$tmp/$2.aux.old"
ok=
for i in $(seq 5); do
latex --halt-on-error \
--output-directory="$tmp" \
--recorder \
"$2.latex" </dev/null
if diff "$tmp/$2.aux.old" \
"$tmp/$2.aux" >/dev/null; then
# .aux file converged, so we're done
ok=1
break
fi
echo
echo "$0: $2.aux changed: try again (try #$i)"
echo
cp "$tmp/$2.aux" "$tmp/$2.aux.old"
done
if [ "$ok" = "" ]; then
echo "$0: fatal: $2.aux did not converge!"
exit 10
fi
# If the newly produced .dvi disappears, we need
# to redo.
redo-ifchange "$tmp/$2.dvi"
# With --recorder, latex produces a list of files
# it used during its run. Let's depend on all of
# them, so if they ever change, we'll redo.
grep ^INPUT "$tmp/$2.fls" |
cut -d' ' -f2 |
xargs redo-ifchange

1
docs/cookbook/latex/discovery.txt Normal file
View file

@ -0,0 +1 @@
It seems that \(E = m c^2\).

252
docs/cookbook/latex/index.md Normal file
View file

@ -0,0 +1,252 @@
### A LaTeX typesetting example
[LaTeX](https://www.latex-project.org/) is a typesetting system that's
especially popular in academia. Among other things, it lets you produce
postscript and pdf files from a set of (mostly text) input files.
LaTeX documents often include images and charts. In our example, we'll show
how to auto-generate a chart for inclusion using an [R script with
ggplot2](https://ggplot2.tidyverse.org/).
To play with this code on your own machine, get the [redo
source code](https://github.com/apenwarr/redo) and look in the
`docs/cookbook/latex/` directory.
### Generating a plot from an R script
First, let's tell redo how to generate our chart. We'll use
the R language, and ask it to plot some of its sample data (the mpg, "miles
per gallon" data set) and save it to an eps (encapsulated postscript) file.
eps files are usually a good format for LaTeX embedded images, because they
scale to any printer or display resolution.
First, let's make an R script that generates a plot:
<pre><code lang='r' src='mpg.R'></code></pre>
And then a .do file to tie that into redo:
<pre><code lang='sh' src='mpg.eps.do'></code></pre>
We can build and view the image:
```shell
$ redo mpg.eps
redo mpg.eps
# View the file on Linux
$ evince mpg.eps
# View the file on MacOS
$ open mpg.eps
```
### Running the LaTeX processor
Here's the first draft of our very important scientific paper:
<pre><code lang='tex' src='paper.latex'></code></pre>
Notice how it refers to the chart from above, `mpg.eps`, and a text file,
`discovery.txt`. Let's create the latter as a static file.
<pre><code lang='tex' src='discovery.txt'></code></pre>
With all the parts of our document in places, we can now compile it directly
using `pdflatex`:
```shell
$ pdflatex paper.latex
This is pdfTeX, Version 3.14159265-2.6-1.40.17 (TeX Live 2016/Debian) (preloaded format=pdflatex)
restricted \write18 enabled.
entering extended mode
...[a lot of unnecessary diagnostic messages]...
Output written on paper.pdf (2 pages, 68257 bytes).
Transcript written on paper.log.
```
But this has a few problems. First of all, it doesn't understand
dependencies; if `mpg.R` changes, it won't know to rebuild `mpg.eps`.
Secondly, the TeX/LaTeX toolchain has an idiosyncracy that means you might
have to rebuild your document more than once. In our example, we generate a
table of contents, but it ends up getting generated *before* processing the
rest of the content in the document, so it's initially blank. As it
continues, LaTeX produces a file called `paper.aux` with a list of the
references needed by the table of contents, and their page numbers. If we
run LaTeX over again, it'll use that to build a proper of table of contents.
Of course, life is not necessarily so easy. Once the table of contents
isn't blank, it might start to push content onto the next page. This will
change all the page numbers! So we'd have to do it one more time. And that
might lead to even more subtle problems, like a reference to page 99
changing to page 100, which pushes a word onto the next page, which changes
some other page number, and so on. Thus, we need a script that will keep
looping, re-running LaTeX until `paper.aux` stabilizes.
The whole script we'll use is below. Instead of running `pdflatex`
directly, we'll use the regular `latex` command, which produces a .dvi
(DeVice Independent) intermediate file which we can later turn into a pdf or
ps file.
LaTeX produces a bunch of clutter files (like `paper.aux`) that can be used
in future runs, but which also make its execution nondeterministic. To
avoid that problem, we tell it to use a temporary `--output-directory` that
we delete and recreate before each build (although we might need to run
`latex` multiple times in one build, to get `paper.aux` to converge).
<pre><code lang='sh' src='default.runtex.do'></code></pre>
### Virtual targets, side effects, and multiple outputs
Why did we call our script `default.runtex.do`? Why not `default.pdf.do` or
`default.dvi.do`, depending what kind of file we ask LaTeX to produce?
The problem is that the `latex` command actually produces several
files in that temporary directory, and we might want to keep them around.
If we name our .do file after only *one* of those outputs, things get messy.
The biggest problem is that redo requires a .do file to write its output to
$3 (or stdout), so that it can guarantee the output gets replaced
atomically. When there is more than one output, at most one file can
be sent to $3; how do you choose which one? Even worse, some programs don't
even have the ability to choose the output filename; for an input of
`paper.latex`, the `latex` command just writes a bunch of files named
`paper.*` directly. You can't ask it to put just one of them in $3.
The easiest way to handle this situation in redo is to use a "virtual
target", which is a target name that doesn't actually get created has a file,
and has only side effects. You've seen these before: when we use `all.do`
or `clean.do`, we don't expect to produce a file named `all` or `clean`. We
expect redo to run a collection of other commands. In `make`, these are
sometimes called ".PHONY rules" because of the way they are declared in a
`Makefile`. But the rules aren't phony, they really are executed; they just
don't produce output. So in redo we call them "virtual."
When we `redo paper.runtex`, it builds our virtual target. There is no
`paper.runtex` file or directory generated. But as a side effect, a
directory named `paper.tmp` is created.
(Side note: it's tempting to name the directory the same as the target. So
we could have a `paper.runtex` directory instead of `paper.tmp`. This is
not inherently a bad idea, but currently redo behaviour is undefined if you
redo-ifchange a directory. Directories are weird. If one file in that
directory disappears, does that mean you "modified" the output by hand?
What if two redo targets modify the same directory? Should we require
scripts to only atomically replace an entire output directory via $3? And
so on. We might carefully define this behaviour eventually, but for now,
it's better to use a separate directory name and avoid the undefined
behaviour.)
### Depending on side effects produced by virtual targets
Next, we want to produce .pdf and .ps files from the collection of files
produced by the `latex` command, particularly `paper.tmp/paper.dvi`. To do
that, we have to bring our files back from the "virtual target" world into
the real world.
Depending on virtual targets is easy; we'll just
`redo-ifchange paper.runtex`. Then we want to materialize `paper.dvi` from
the temporary files in `paper.tmp/paper.dvi`, which we can do with an
efficient [hardlink](https://en.wikipedia.org/wiki/Hard_link) (rather than
making an unnecessary copy), like this:
<pre><code lang='sh' src='default.dvi.do'></code></pre>
Notice that we *don't* do `redo-ifchange paper.tmp/paper.dvi`. That's
because redo has no knowledge of that file. If you ask redo to build that
file for you, it doesn't know how to do it. You have to ask for
`paper.runtex`, which you know - but redo doesn't know - will produce the
input file you want. Then you can safely use it.
Once we have a .do file that produces the "real" (non-virtual,
non-side-effect) `paper.dvi` file, however, it's safe to depend directly on
it. Let's use that to produce our .ps and .pdf outputs:
<pre><code lang='sh' src='default.ps.do'></code></pre>
<pre><code lang='sh' src='default.pdf.do'></code></pre>
(As above, we include `exec >&2` lines because LaTeX tools incorrectly write
their log messages to stdout. We need to redirect it all to stderr. That
way [redo-log](../../redo-log) can handle all the log output appropriately.)
### Explicit dependencies
We've made a generalized script, `default.runtex.do`, that can compile any
.latex file and produce a .tmp directory with its output. But that's not
quite enough: different .latex files might have extra dependencies that need
to exist *before* the compilation can continue. In our case, we need the
auto-generated `mpg.eps` that we discussed above.
To make that work, `default.runtex.do` looks for a .deps file with the same
name as the .latex file being processed. It contains just a list of extra
dependencies that need to be built. Here's ours:
<pre><code src='paper.deps'></code></pre>
You can use this same ".deps" technique in various different places in redo.
For example, you could have a default.do that can link a C program from any
set of .o files. To specify the right set of .o files for target `X`,
default.do might look in an `X.deps` or `X.list` file. If you later want to
get even fancier, you could make an `X.deps.do` that programmatically
generates the list of dependencies; for example, it might include one set of
files on win32 platforms and a different set on unix platforms.
### Autodependencies
Our `paper.latex` file actually includes two files: `mpg.eps`, which we
explicitly depended upon above, and `discovery.txt`, which we didn't. The
latter is a static source file, so we can let redo discover it
automatically, based on the set of files that LaTeX opens while it runs.
The `latex` command has a `--record` option to do this; it produces a file
called `paper.tmp/paper.fls` (.fls is short for "File LiSt").
One of redo's best features is that you can declare dependencies *after*
you've done your build steps, when you have the best knowledge of which
files were actually needed. That's why in `default.runtex.do`, we parse the
.fls file and then redo-ifchange on its contents right at the end.
(This brings up a rather subtle point about how redo works. When you run
redo-ifchange, redo adds to the list of files which, if they change, mean
your target needs to be rebuilt. But unlike make, redo will not actually
rebuild those files merely because they're listed as a dependency; it just
knows to rebuild your target, which means to run your .do file, which will
run redo-ifchange *again* if it still needs those input files to be fresh.
This avoids an annoying problem in `make` where you can teach it about
which .h files your C program depended on last time, but if you change
A.c to no longer include X.h, and then delete X.h, make might complain
that X.h is missing, because A.c depended on it *last time*. redo will
simply notice that since X.h is missing, A.c needs to be recompiled, and let
your compilation .do script report an error, or not.)
Anyway, this feature catches not just our `discovery.txt` dependency, but
also the implicit dependencies on various LaTeX template and font files, and
so on. If any of those change, our LaTeX file needs to be rebuilt.
```shell
$ redo --no-detail paper.pdf
redo paper.pdf
redo paper.dvi
redo paper.runtex
redo mpg.eps
$ redo --no-detail paper.pdf
redo paper.pdf
$ touch discovery.txt
$ redo --no-detail paper.pdf
redo paper.pdf
redo paper.dvi
redo paper.runtex
$ redo --no-detail paper.pdf
redo paper.pdf
```
### Housekeeping
As usual, to polish up our project, let's create an `all.do` and
`clean.do`.
Because this project is included in the redo source and we don't want redo
to fail to build just because you don't have LaTeX or R installed, we'll
have `all.do` quit politely if the necessary tools are missing.
<pre><code lang='sh' src='all.do'></code></pre>
<pre><code lang='sh' src='clean.do'></code></pre>

4
docs/cookbook/latex/mpg.R Normal file
View file

@ -0,0 +1,4 @@
library(ggplot2)
qplot(mpg, wt, data = mtcars) + facet_wrap(~cyl) + theme_bw()
ggsave("mpg.new.eps", width=4, height=2, units='in')

7
docs/cookbook/latex/mpg.eps.do Normal file
View file

@ -0,0 +1,7 @@
redo-ifchange mpg.R
Rscript mpg.R >&2
mv mpg.new.eps $3
# Some buggy ggplot2 versions produce this
# junk file; throw it away.
rm -f Rplots.pdf

1
docs/cookbook/latex/paper.deps Normal file
View file

@ -0,0 +1 @@
mpg.eps

17
docs/cookbook/latex/paper.latex Normal file
View file

@ -0,0 +1,17 @@
\documentclass{article}
\usepackage{graphicx}
\title{A very brief note on relativity}
\author{The Redo Contributors}
\begin{document}
\maketitle
\tableofcontents
\newpage
\section{Amazing Discovery}
\input{discovery.txt}
\section{Irrelevant Chart}
\includegraphics{mpg.eps}
\end{document}

2
docs/default.1.do Normal file
View file

@ -0,0 +1,2 @@
redo-ifchange md-to-man $2.md.tmp
. ./md-to-man $1 $2 $3

7
docs/default.md.tmp.do Normal file
View file

@ -0,0 +1,7 @@
redo-ifchange ../redo/version/vars $2.md
. ../redo/version/vars
cat - $2.md <<-EOF
% $2(1) Redo $TAG
% Avery Pennarun <apenwarr@gmail.com>
% $DATE
EOF

3
docs/doc.list.do Normal file
View file

@ -0,0 +1,3 @@
ls redo*.md t/*.md >$3
redo-always
redo-stamp <$3

8
docs/extra_style.css Normal file
View file

@ -0,0 +1,8 @@
p code {
border: 0;
padding: 0 2px 0 2px;
font-size: 100%;
white-space: nowrap;
color: #a04040;
letter-spacing: -1px;
}

25
docs/fetchcode.js Normal file
View file

@ -0,0 +1,25 @@
'use strict';
async function fetchCode(e, lang, src) {
const resp = await fetch(src);
let t = await resp.text();
if (lang) t = hljs.highlight(lang, t).value;
e.innerHTML = t;
}
function fetchAndHighlightAll() {
const el = document.getElementsByTagName('code');
for (const e of el) {
const src = e.getAttribute('src');
if (!src) continue;
const lang = e.getAttribute('lang');
const title = document.createElement('b');
title.innerText = src;
title.style.textAlign = 'center';
title.style.display = 'block';
e.parentElement.insertBefore(title, e);
fetchCode(e, lang, src);
}
}
setTimeout(fetchAndHighlightAll, 0);

15
docs/git-export.do Normal file
View file

@ -0,0 +1,15 @@
# update the local 'man' branch with pregenerated output files, for people
# who don't have pandoc (and maybe to aid in google searches or something)
redo-ifchange all
git update-ref refs/heads/man origin/man '' 2>/dev/null || true
export GIT_INDEX_FILE=gitindex.tmp
rm -f $GIT_INDEX_FILE
git add -f *.1
MSG="Autogenerated man pages for $(git describe)"
TREE=$(git write-tree --prefix=docs)
git show-ref refs/heads/man >/dev/null && PARENT="-p refs/heads/man"
COMMITID=$(echo "$MSG" | git commit-tree $TREE $PARENT)
git update-ref refs/heads/man $COMMITID

8
docs/git-import.do Normal file
View file

@ -0,0 +1,8 @@
exec >&2
if git show-ref refs/heads/man >/dev/null; then
(cd .. && git archive man) | tar -xvf -
else
(cd .. && git archive origin/man) | tar -xvf -
fi

258
docs/index.md Normal file
View file

@ -0,0 +1,258 @@
# redo: a recursive, general-purpose build system
`redo` is a competitor to the long-lived, but sadly imperfect, `make`
program. Unlike other such competitors, redo captures the essential
simplicity and flexibility of make, while avoiding its flaws. It manages to
do this while being simultaneously simpler than make, more flexible than
make, and more powerful than make, and without sacrificing performance - a
rare combination of features.
The original design for redo comes from Daniel J. Bernstein (creator of
qmail and djbdns, among many other useful things). He posted some
terse notes on his web site at one point (there is no date) with the
unassuming title, "[Rebuilding target files when source files have
changed](http://cr.yp.to/redo.html)." Those notes are enough information to
understand how the system is supposed to work; unfortunately there's no code
to go with it. I wrote this implementation of redo from scratch, based on
that design.
After I found out about djb redo, I searched the Internet for any sign that
other people had discovered what I had: a hidden, unimplemented gem of
brilliant code design. I found only one interesting link: Alan Grosskurth,
whose [Master's thesis at the University of Waterloo](http://grosskurth.ca/papers/mmath-thesis.pdf)
was about top-down software rebuilding, that is, djb redo. He wrote his
own (admittedly slow) implementation in about 250 lines of shell script,
which gives an idea for how straightforward the system is.
My implementation of redo is called `redo` for the same reason that there
are 75 different versions of `make` that are all called `make`. It's somehow
easier that way.
I also provide an extremely minimal pure-POSIX-sh implementation, called
`do`, in the `minimal/` directory of this repository.
(Want to discuss redo? See the bottom of this file for
information about our mailing list.)
# What's so special about redo?
The theory behind redo sounds too good to be true: it can do everything
`make` can do, but the implementation is vastly simpler, the syntax is
cleaner, and you have even more flexibility without resorting to ugly hacks.
Also, you get all the speed of non-recursive `make` (only check dependencies
once per run) combined with all the cleanliness of recursive `make` (you
don't have code from one module stomping on code from another module).
(Disclaimer: my current implementation is not as fast as `make` for some
things, because it's written in python. Eventually I'll rewrite it an C and
it'll be very, very fast.)
The easiest way to show it is to jump into an example. Here's one for
compiling a C++ program.
Create a file called default.o.do:
redo-ifchange $2.c
gcc -MD -MF $2.d -c -o $3 $2.c
read DEPS <$2.d
redo-ifchange ${DEPS#*:}
Create a file called myprog.do:
DEPS="a.o b.o"
redo-ifchange $DEPS
gcc -o $3 $DEPS
Of course, you'll also have to create `a.c` and `b.c`, the C language
source files that you want to build to create your application.
In a.c:
#include <stdio.h>
#include "b.h"
int main() { printf(bstr); }
In b.h:
extern char *bstr;
In b.c:
char *bstr = "hello, world!\n";
Now you simply run:
$ redo myprog
And it says:
redo myprog
redo a.o
redo b.o
Now try this:
$ touch b.h
$ redo myprog
Sure enough, it says:
redo myprog
redo a.o
Did you catch the shell incantation in `default.o.do` where it generates
the autodependencies? The filename `default.o.do` means "run this script to
generate a .o file unless there's a more specific whatever.o.do script that
applies."
The key thing to understand about redo is that declaring a dependency is just
another shell command. The `redo-ifchange` command means, "build each of my
arguments. If any of them or their dependencies ever change, then I need to
run the *current script* over again."
Dependencies are tracked in a persistent `.redo` database so that redo can
check them later. If a file needs to be rebuilt, it re-executes the
`whatever.do` script and regenerates the dependencies. If a file doesn't
need to be rebuilt, redo figures that out just using its persistent
`.redo` database, without re-running the script. And it can do that check
just once right at the start of your project build, which is really fast.
Best of all, as you can see in `default.o.do`, you can declare a dependency
*after* building the program. In C, you get your best dependency
information by trying to actually build, since that's how you find out which
headers you need. redo is based on this simple insight: you don't
actually care what the dependencies are *before* you build the target. If
the target doesn't exist, you obviously need to build it.
Once you're building it anyway, the build script itself can calculate the
dependency information however it wants; unlike in `make`, you don't need a
special dependency syntax at all. You can even declare some of your
dependencies after building, which makes C-style autodependencies much
simpler.
redo therefore is a unique combination of imperative and declarative
programming. The initial build is almost entirely imperative (running a
series of scripts). As part of that, the scripts declare dependencies a few
at a time, and redo assembles those into a larger data structure. Then, in
the future, it uses that pre-declared data structure to decide what work
needs to be redone.
(GNU make supports putting some of your dependencies in include files, and
auto-reloading those include files if they change. But this is very
confusing - the program flow through a Makefile is hard to trace already,
and even harder when it restarts from the beginning because an include file
changes at runtime. With redo, you can just read each build script from top
to bottom. A `redo-ifchange` call is like calling a function, which you can
also read from top to bottom.)
# What projects use redo?
Some larger proprietary projects are using it, but unfortunately they can't
easily be linked from this document. Here are a few open source examples:
* [Liberation Circuit](https://github.com/linleyh/liberation-circuit) is a
straightforward example of a C++ binary (a game) compiled with redo.
* [WvStreams](https://github.com/apenwarr/wvstreams) uses a more complex
setup producing several binaries, libraries, and scripts. It shows how to
produce output files in a different directory than the source files.
* [WvBuild](https://github.com/apenwarr/wvbuild) can cross-compile several
dependencies, like openssl and zlib, and then builds WvStreams using those
same libraries. It's a good example of redo/make interop and complex
dependencies.
* There's an experimental [variant of
Buildroot](https://github.com/apenwarr/buildroot/tree/redo) that uses redo
in order to clean up its dependency logic.
* You can also find some curated examples in the
[docs/cookbook/](https://github.com/apenwarr/redo/tree/master/docs/cookbook/) subdir of the redo project itself.
* A [github search for all.do](https://github.com/search?p=9&q=path%3A%2F+extension%3Ado+filename%3A%2Fall.do&type=Code)
shows an ever-growing number of projects using redo.
If you switch your program's build process to use redo, please let us know and
we can link to it here for some free publicity.
(Please don't use the integration testing code in the redo project's `t/`
directory as serious examples of how to use redo. Many of the tests are
doing things in intentionally psychotic ways in order to stress redo's code
and find bugs.)
# How does this redo compare to other redo implementations?
djb never released his version, so other people have implemented their own
variants based on his [published specification](http://cr.yp.to/redo.html).
This version, sometimes called apenwarr/redo, is probably the most advanced
one, including support for parallel builds,
[resilient timestamps](https://apenwarr.ca/log/20181113) and checksums,
[build log linearization](https://apenwarr.ca/log/20181106), and
helpful debugging features. It's currently written in python for easier
experimentation, but the plan is to eventually migrate it to plain C. (Some
people like to call this version "python-redo", but I don't like that name.
We shouldn't have to rename it when we later transliterate the code to C.)
Here are some other redo variants (thanks to Nils Dagsson Moskopp for many
of these links):
- Alan Grosskurth's [redo thesis](http://grosskurth.ca/papers/mmath-thesis.pdf)
and related sh implementation. (Arguably, this paper is the one that got
all the rest of us started.)
- Nils Dagsson Moskopp's [redo-sh](https://web.archive.org/web/20181106195145/http://news.dieweltistgarnichtso.net/bin/redo-sh.html)
is a completely self-sufficient sh-based implementation.
- apenwarr's [minimal/do](https://github.com/apenwarr/redo/blob/master/minimal/do)
is included with this copy of redo. It's also sh-based, but intended to
be simple and failsafe, so it doesn't understand how to "redo" targets more
than once.
- Christian Neukirchen's [redo-c](https://github.com/chneukirchen/redo-c), a
C implementation.
- Jonathan de Boyne Pollard's [fork of Alan Grosskurth's redo](http://jdebp.eu./Softwares/redo/grosskurth-redo.html)
(another sh-based implementation).
- Jonathan de Boyne Pollard's [redo](http://jdebp.eu./Softwares/redo/)
rewritten in C++
- Gyepi Sam's [redux](https://github.com/gyepisam/redux) in Go
- jekor's [redo](https://github.com/jekor/redo) in Haskell
- Shanti Bouchez-Mongardé (mildred)'s [fork of apenwarr's redo](https://github.com/mildred/redo)
in python
- Tharre's [redo](https://github.com/Tharre/redo) in C
- catenate's [credo](https://github.com/catenate/credo), a (very
rearchitected) variant written for the Inferno Shell.
The original redo design is so simple and elegant that many individuals
have been
inspired to (and able to) write their own version of it. In the honoured
tradition of Unix's `make`, they (almost) all just use the same name,
`redo`. Unfortunately, many of these
implementations are unmaintained, slightly incompatible with the "standard"
redo semantics, and/or have few or no automated tests.
At the time of this writing, none of them except apenwarr/redo (ie. this
project) support parallel builds (`redo -j`) or log linearization
(`redo-log`). For large projects, parallel builds are usually considered
essential.
The [automated tests](https://github.com/apenwarr/redo/tree/master/t) in
this version of redo are almost, but not quite, appropriate for testing any
redo implementation. If you really must write a new version of redo, we
invite you to thoroughly test it against the existing test suite to ensure
compatibility. You can also steal our tests (with attribution, of course)
and include them in your own source package. We'd also love it it you
contribute more automated tests when you find a bug, or send us patches if
you find a test which is accidentally incompatible (as opposed to finding a
real bug) with other redo implementations.

9
docs/md-to-man.do Normal file
View file

@ -0,0 +1,9 @@
redo-ifchange md2man.py ../redo/whichpython
read py <../redo/whichpython
if ../redo/python ./md2man.py /dev/null /dev/null >/dev/null; then
echo '../redo/python ./md2man.py $2.md.tmp $2.html'
else
echo "Warning: md2man.py missing modules; can't generate manpages." >&2
echo "Warning: try this: sudo easy_install markdown BeautifulSoup" >&2
echo 'echo Skipping: $2.1 >&2'
fi

282
docs/md2man.py Normal file
View file

@ -0,0 +1,282 @@
import sys, os, markdown, re
from BeautifulSoup import BeautifulSoup
def _split_lines(s):
return re.findall(r'([^\n]*\n?)', s)
class Writer:
def __init__(self):
self.started = False
self.indent = 0
self.last_wrote = '\n'
def _write(self, s):
if s:
self.last_wrote = s
sys.stdout.write(s)
def writeln(self, s):
if s:
self.linebreak()
self._write('%s\n' % s)
def write(self, s):
if s:
self.para()
for line in _split_lines(s):
if line.startswith('.'):
self._write('\\&' + line)
else:
self._write(line)
def linebreak(self):
if not self.last_wrote.endswith('\n'):
self._write('\n')
def para(self, bullet=None):
if not self.started:
if not bullet:
bullet = ' '
if not self.indent:
self.writeln(_macro('.PP'))
else:
assert(self.indent >= 2)
prefix = ' '*(self.indent-2) + bullet + ' '
self.writeln('.IP "%s" %d' % (prefix, self.indent))
self.started = True
def end_para(self):
self.linebreak()
self.started = False
def start_bullet(self):
self.indent += 3
self.para(bullet='\\[bu]')
def end_bullet(self):
self.indent -= 3
self.end_para()
w = Writer()
def _macro(name, *args):
if not name.startswith('.'):
raise ValueError('macro names must start with "."')
fixargs = []
for i in args:
i = str(i)
i = i.replace('\\', '')
i = i.replace('"', "'")
if (' ' in i) or not i:
i = '"%s"' % i
fixargs.append(i)
return ' '.join([name] + list(fixargs))
def macro(name, *args):
w.writeln(_macro(name, *args))
def _force_string(owner, tag):
if tag.string:
return tag.string
else:
out = ''
for i in tag:
if not (i.string or i.name in ['a', 'br']):
raise ValueError('"%s" tags must contain only strings: '
'got %r: %r' % (owner.name, tag.name, tag))
out += _force_string(owner, i)
return out
def _clean(s):
s = s.replace('\\', '\\\\')
return s
def _bitlist(tag):
if getattr(tag, 'contents', None) == None:
for i in _split_lines(str(tag)):
yield None,_clean(i)
else:
for e in tag:
name = getattr(e, 'name', None)
if name in ['a', 'br']:
name = None # just treat as simple text
s = _force_string(tag, e)
if name:
yield name,_clean(s)
else:
for i in _split_lines(s):
yield None,_clean(i)
def _bitlist_simple(tag):
for typ,text in _bitlist(tag):
if typ and not typ in ['em', 'strong', 'code']:
raise ValueError('unexpected tag %r inside %r' % (typ, tag.name))
yield text
def _text(bitlist):
out = ''
for typ,text in bitlist:
if not typ:
out += text
elif typ == 'em':
out += '\\fI%s\\fR' % text
elif typ in ['strong', 'code']:
out += '\\fB%s\\fR' % text
else:
raise ValueError('unexpected tag %r inside text' % (typ,))
out = out.strip()
out = re.sub(re.compile(r'^\s+', re.M), '', out)
return out
def text(tag):
w.write(_text(_bitlist(tag)))
# This is needed because .BI (and .BR, .RB, etc) are weird little state
# machines that alternate between two fonts. So if someone says something
# like foo<b>chicken</b><b>wicken</b>dicken we have to convert that to
# .BI foo chickenwicken dicken
def _boldline(l):
out = ['']
last_bold = False
for typ,text in l:
nonzero = not not typ
if nonzero != last_bold:
last_bold = not last_bold
out.append('')
out[-1] += re.sub(r'\s+', ' ', text)
macro('.BI', *out)
def do_definition(tag):
w.end_para()
macro('.TP')
w.started = True
split = 0
pre = []
post = []
for typ,text in _bitlist(tag):
if split:
post.append((typ,text))
elif text.lstrip().startswith(': '):
split = 1
post.append((typ,text.lstrip()[2:].lstrip()))
else:
pre.append((typ,text))
_boldline(pre)
w.write(_text(post))
w.started = False
def do_list(tag):
for i in tag:
name = getattr(i, 'name', '').lower()
if not name and not str(i).strip():
pass
elif name != 'li':
raise ValueError('only <li> is allowed inside <ul>: got %r' % i)
else:
w.start_bullet()
for xi in i:
if str(xi).strip():
do(xi)
w.end_para()
w.end_bullet()
def do(tag):
name = getattr(tag, 'name', '').lower()
if not name:
text(tag)
elif name == 'h1':
macro('.SH', _force_string(tag, tag).upper())
w.started = True
elif name == 'h2':
macro('.SS', _force_string(tag, tag))
w.started = True
elif name.startswith('h') and len(name)==2:
raise ValueError('%r invalid - man page headers must be h1 or h2'
% name)
elif name == 'pre':
t = _force_string(tag.code, tag.code)
if t.strip():
macro('.RS', '+4n')
macro('.nf')
w.write(_clean(t).rstrip())
macro('.fi')
macro('.RE')
w.end_para()
elif name == 'p' or name == 'br':
g = re.match(re.compile(r'([^\n]*)\n *: +(.*)', re.S), str(tag))
if g:
# it's a definition list (which some versions of python-markdown
# don't support, including the one in Debian-lenny, so we can't
# enable that markdown extension). Fake it up.
do_definition(tag)
else:
text(tag)
w.end_para()
elif name == 'ul':
do_list(tag)
else:
raise ValueError('non-man-compatible html tag %r' % name)
PROD='Untitled'
VENDOR='Vendor Name'
SECTION='9'
GROUPNAME='User Commands'
DATE=''
AUTHOR=''
lines = []
if len(sys.argv) != 3:
sys.stderr.write('usage: %s <infile.md> <outfile.html> >outfile.1\n')
sys.exit(99)
infile = sys.argv[1]
htmlfile = sys.argv[2]
lines += open(infile).read().decode('utf8').split('\n')
# parse pandoc-style document headers (not part of markdown)
g = re.match(r'^%\s+(.*?)\((.*?)\)\s+(.*)$', lines[0])
if g:
PROD = g.group(1)
SECTION = g.group(2)
VENDOR = g.group(3)
lines.pop(0)
g = re.match(r'^%\s+(.*?)$', lines[0])
if g:
AUTHOR = g.group(1)
lines.pop(0)
g = re.match(r'^%\s+(.*?)$', lines[0])
if g:
DATE = g.group(1)
lines.pop(0)
g = re.match(r'^%\s+(.*?)$', lines[0])
if g:
GROUPNAME = g.group(1)
lines.pop(0)
inp = '\n'.join(lines)
if AUTHOR:
inp += ('\n# AUTHOR\n\n%s\n' % AUTHOR).replace('<', '&lt;')
html = markdown.markdown(inp)
open(htmlfile, 'w').write(html)
soup = BeautifulSoup(html, convertEntities=BeautifulSoup.HTML_ENTITIES)
macro('.TH', PROD.upper(), SECTION, DATE, VENDOR, GROUPNAME)
macro('.ad', 'l') # left justified
macro('.nh') # disable hyphenation
for e in soup:
do(e)

9
docs/mkdocs.do Normal file
View file

@ -0,0 +1,9 @@
redo-ifchange doc.list
xargs redo-ifchange ../mkdocs.yml <doc.list
if type mkdocs >/dev/null 2>/dev/null; then
(cd .. && mkdocs build)
else
echo "Warning: mkdocs is missing; can't generate website." >&2
redo-ifcreate /usr/bin/mkdocs
fi

49
docs/redo-always.md Normal file
View file

@ -0,0 +1,49 @@
# NAME
redo-always - mark the current target as always needing to be rebuilt
# SYNOPSIS
redo-always
# DESCRIPTION
Normally redo-always is run from a .do file that has been
executed by `redo`(1). See `redo`(1) for more details.
redo-always takes no parameters. It simply adds an
'impossible' dependency to the current target, which
ensures that the target will always be rebuilt if anyone
runs `redo-ifchange targetname`.
Because of the way redo works, `redo-ifchange targetname`
will only rebuild `targetname` once per session. So if
multiple targets depend on *targetname* and *targetname*
has called redo-always, only the first target will cause it
to be rebuilt. If the build cycle completes and a new one
begins, it will be rebuilt exactly one more time.
Normally, any target that depends (directly or indirectly)
on a sub-target that has called redo-always will also
always need to rebuild, since one of its dependencies will
always be out of date. To avoid this problem, redo-always is
usually used along with `redo-stamp`(1).
# REDO
Part of the `redo`(1) suite.
# CREDITS
The original concept for `redo` was created by D. J.
Bernstein and documented on his web site
(http://cr.yp.to/redo.html). This independent implementation
was created by Avery Pennarun and you can find its source
code at http://github.com/apenwarr/redo.
# SEE ALSO
`redo`(1), `redo-ifcreate`(1), `redo-ifchange`(1), `redo-stamp`(1)

92
docs/redo-ifchange.md Normal file
View file

@ -0,0 +1,92 @@
# NAME
redo-ifchange - rebuild target files when source files have changed
# SYNOPSIS
redo-ifchange [targets...]
# DESCRIPTION
Normally redo-ifchange is run from a .do file that has been
executed by `redo`(1). See `redo`(1) for more details.
redo-ifchange doesn't take any command line options other
than a list of *targets*. To provide command line options,
you need to run `redo` instead.
redo-ifchange performs the following steps:
- it creates a dependency on the given *targets*. If any
of those targets change in the future, the current target
(the one calling redo-ifchange) will marked as needing to
be rebuilt.
- for any *target* that is out of date, it calls the
equivalent of `redo target`.
- for any *target* that is locked (because some other
instance of `redo` or `redo-ifchange` is already building
it), it waits until the lock is released.
redo-ifchange returns only after all the given
*targets* are known to be up to date.
# TIP 1
You don't have to run redo-ifchange *before* generating
your target; you can generate your target first, then
declare its dependencies. For example, as part of
compiling a .c file, gcc learns the list
of .h files it depends on. You can pass this information
along to redo-ifchange, so if any of those headers are
changed or deleted, your .c file will be rebuilt:
redo-ifchange $2.c
gcc -o $3 -c $2.c \
-MMD -MF $2.deps
read DEPS <$2.deps
redo-ifchange ${DEPS#*:}
This is much less confusing than the equivalent
autodependency mechanism in `make`(1), because make
requires that you declare all your dependencies before
running the target build commands.
# TIP 2
Try to list as many dependencies as possible in a single
call to redo-ifchange. Every time you run redo-ifchange,
the shell has to fork+exec it, which takes time. Plus redo
can only parallelize your build if you give it multiple
targets to build at once. It's fine to have a couple of
separate redo-ifchange invocations for a particular target
when necessary (as in TIP 1 above), but try to keep it to a
minimum. For example here's a trick for generating a list
of targets, but redo-ifchanging them all at once:
for d in *.c; do
echo ${d%.c}.o
done |
xargs redo-ifchange
# REDO
Part of the `redo`(1) suite.
# CREDITS
The original concept for `redo` was created by D. J.
Bernstein and documented on his web site
(http://cr.yp.to/redo.html). This independent implementation
was created by Avery Pennarun and you can find its source
code at http://github.com/apenwarr/redo.
# SEE ALSO
`redo`(1), `redo-ifcreate`(1), `redo-always`(1), `redo-stamp`(1)

43
docs/redo-ifcreate.md Normal file
View file

@ -0,0 +1,43 @@
# NAME
redo-ifcreate - rebuild the current target if source files are created
# SYNOPSIS
redo-ifcreate [sources...]
# DESCRIPTION
Normally redo-ifcreate is run from a .do file that has been
executed by `redo`(1). See `redo`(1) for more details.
redo-ifcreate takes a list of nonexistent files (*sources*)
and adds them as dependencies to the current target (the
one calling redo-ifcreate). If any of those files are
created in the future, the target will be marked as needing
to be rebuilt.
If one of the given files exists at the time redo-ifcreate
is called, it will return a nonzero exit code.
If you want to declare dependencies on files that already
exist, use `redo-ifchange`(1) instead.
# REDO
Part of the `redo`(1) suite.
# CREDITS
The original concept for `redo` was created by D. J.
Bernstein and documented on his web site
(http://cr.yp.to/redo.html). This independent implementation
was created by Avery Pennarun and you can find its source
code at http://github.com/apenwarr/redo.
# SEE ALSO
`redo`(1), `redo-ifchange`(1), `redo-always`(1), `redo-stamp`(1)

83
docs/redo-log.md Normal file
View file

@ -0,0 +1,83 @@
# NAME
redo-log - display log messages from previous builds
# SYNOPSIS
redo-log [options...] [targets...]
# DESCRIPTION
When redo runs, it captures the standard error (stderr) output from the
build activity for each target, and saves it for later. When a target is
rebuilt, the new logs replace the old logs for that target. redo-log
prints the log output for any set of targets.
# OPTIONS
-r, --recursive
: if the requested targets cause any dependencies to be built, recursively
show the logs from those dependencies as well. (And if those
dependencies build further dependencies, also show those logs, and so on.)
-u, --unchanged
: show messages even for dependencies that were unchanged (did not need to be
rebuilt). To do this, we show the logs for the *most recent* build of
each affected dependency. Usually this is used with `-r`.
-f, --follow
: if a build is currently running for any of the requested targets or
their dependencies, follow the logs (like `tail -f`) until the build
finishes.
--no-details
: display *only* the messages from redo itself, not the other messages
produced by build scripts. Generally this gives you a list of which
targets were built, but not detailed logs, warnings, or errors.
--no-status
: don't display the running build status at the bottom of the screen.
(Unless this option is specified, the status line will be enabled
if using --follow, if stderr is a terminal.) If stderr is *not* a
terminal, you can force enable the status line using --status.
--no-pretty
: display "raw" redo log lines (@@REDO events) rather than using a
human-readable format. The default is --pretty.
--no-color
: when using --pretty and writing to a terminal, colorize the output to
make results stand out more clearly. If not writing to a terminal, you
can use --color to force colorized output.
--debug-locks
: print messages about acquiring, releasing, and waiting
on locks. Because redo can be highly parallelized,
one instance may end up waiting for a target to be
built by some other instance before it can continue.
If you suspect this is causing troubles, use this
option to see which instance is waiting and when.
--debug-pids
: add the process id of the particular redo instance to each
output message. This makes it easier to figure out
which sub-instance of redo is doing what.
# REDO
Part of the `redo`(1) suite.
# CREDITS
The original concept for `redo` was created by D. J.
Bernstein and documented on his web site
(http://cr.yp.to/redo.html). This independent implementation
was created by Avery Pennarun and you can find its source
code at http://github.com/apenwarr/redo.
# SEE ALSO
`redo`(1)

55
docs/redo-ood.md Normal file
View file

@ -0,0 +1,55 @@
# NAME
redo-ood - print the list of out-of-date redo targets
# SYNOPSIS
redo-ood
# DESCRIPTION
redo-ood prints a list of all redo *target* files that
exist, but are out of date.
Files that no longer exist might not be targets anymore;
you'll have to redo them for them to end up back in this
list. (For example, if you built a file and then removed
the file and its .do file, you wouldn't want it to show up
in this list.)
If a .do script does not produce an output file (eg.
all.do, clean.do), it also does not show up in this list.
Each filename is on a separate line. The filenames are not
guaranteed to be in any particular order.
All filenames are printed relative the current directory.
The list is not filtered in any way; it contains *all* the
target filenames from the entire project. Remember that
the redo database may span more than just your project, so
you might need to filter the list before using it. (A
useful heuristic might be to remove any line starting
with '../' since it often refers to a target you don't care
about.)
If you want a list of all targets, not just out-of-date
ones, use `redo-targets`(1).
# REDO
Part of the `redo`(1) suite.
# CREDITS
The original concept for `redo` was created by D. J.
Bernstein and documented on his web site
(http://cr.yp.to/redo.html). This independent implementation
was created by Avery Pennarun and you can find its source
code at http://github.com/apenwarr/redo.
# SEE ALSO
`redo`(1), `redo-targets`(1), `redo-sources`(1)

48
docs/redo-sources.md Normal file
View file

@ -0,0 +1,48 @@
# NAME
redo-sources - print the list of all known redo sources
# SYNOPSIS
redo-sources
# DESCRIPTION
redo-sources prints a list of all redo *source* files that
still exist.
A source file is any file that has been listed as a
dependency (using `redo-ifchange`(1) or `redo-ifcreate`(1))
but is not itself a target. A target is a file that
`redo`(1) can build using a .do script.
Each filename is on a separate line. The filenames are not
guaranteed to be in any particular order.
All filenames are printed relative the current directory.
The list is not filtered in any way; it contains *all* the
source filenames from the entire project. Remember that
the redo database may span more than just your project, so
you might need to filter the list before using it.
If you want a list of targets instead of sources, use
`redo-targets`(1) or `redo-ood`(1).
# REDO
Part of the `redo`(1) suite.
# CREDITS
The original concept for `redo` was created by D. J.
Bernstein and documented on his web site
(http://cr.yp.to/redo.html). This independent implementation
was created by Avery Pennarun and you can find its source
code at http://github.com/apenwarr/redo.
# SEE ALSO
`redo`(1), `redo-targets`(1), `redo-ood`(1)

110
docs/redo-stamp.md Normal file
View file

@ -0,0 +1,110 @@
# NAME
redo-stamp - detect if the current target has meaningfully changed
# SYNOPSIS
redo-stamp <$3
# DESCRIPTION
Normally, when `redo`(1) builds a target T, all the other
targets that depend on T are marked as out of date. Even
if the rebuilt T is identical to the old one, all its
dependents need to be rebuilt.
By calling redo-stamp from your .do script, you can tell
`redo` that even though the current target is building, its
output may turn out to be unchanged. If it hasn't, `redo`
may then be able to avoid building other targets that
depend on this target.
redo-stamp marks the current target as changed or unchanged
by comparing its stdin to the input that was provided last
time redo-stamp was called for this target.
The stamp data can be anything you want. Some possibilities
are:
- the actual target file contents:
redo-stamp <$3
- a list of filenames:
find -name '*.[ch]' | sort | redo-stamp
- the contents of a web page:
curl http://example.org | redo-stamp
To ensure that your target gets checked every time, you
might want to use `redo-always`(1).
# DISCUSSION
While using redo-stamp is simple, the way it
works is harder to explain. Deciding if a target is
up to date or not is the job of `redo-ifchange`(1).
Normally, a target is considered out of date when any of its
dependencies (direct or indirect) have changed. When that
happens, `redo-ifchange` runs the .do script for the
target, which regenerates the entire dependency list,
including rebuilding any dependencies as necessary.
Imagine that we have the following dependency chain:
T -> X -> Y -> Z
T depends on X, which depends on Y, which depends
on Z. Now imagine someone has changed Z.
If someone runs `redo-ifchange T`, then redo-ifchange
checks if X is up to date; to do that, it checks if Y
is up to date; and to do that, it checks whether Z is up to
date - which it isn't. Thus, Y is not up to date, which
means X isn't, which means T isn't either, and so we need
to run T.do. `redo-ifchange` won't directly `redo X` just
because X is dirty; it redoes T, and T.do might eventually
call `redo-ifchange X` if it needs to.
When using redo-stamp, the behaviour is different. Let's
say Y has used redo-stamp. In the above sequence, Y is
outdated because Z has changed. However, we don't know yet
if Y's stamp has changed, so we don't yet know if we need
to redo X, and thus we don't know if we need to redo T. We
can't just run `redo T`, because that could waste a lot of
time if it turns out T didn't need to be rebuilt after all.
What we do instead is note whether the *only* dependencies
of T that are out of date are 'stamped' targets. If *any*
dependencies of T are normal, out-of-date redo targets,
then T needs to be rebuilt anyway; we don't have to do
anything special.
If the only dependency of T that has changed is Y (a
'stamped' target), then we need to `redo Y` automatically
in order to determine if T needs to be rebuilt. This is
the only time that `redo` ever rebuilds a target that
hasn't been explicitly asked for as part of a running .do
script.
# REDO
Part of the `redo`(1) suite.
# CREDITS
The original concept for `redo` was created by D. J.
Bernstein and documented on his web site
(http://cr.yp.to/redo.html). This independent implementation
was created by Avery Pennarun and you can find its source
code at http://github.com/apenwarr/redo.
# SEE ALSO
`redo`(1), `redo-ifcreate`(1), `redo-ifchange`(1), `redo-always`(1)

60
docs/redo-targets.md Normal file
View file

@ -0,0 +1,60 @@
# NAME
redo-targets - print the list of all known redo targets
# SYNOPSIS
redo-targets
# DESCRIPTION
redo-targets prints a list of all redo *target* files that
still exist.
Files that no longer exist might not be targets anymore;
you'll have to redo them for them to end up back in this
list. (For example, if you built a file and then removed
the file and its .do file, you wouldn't want it to show up
in this list.)
If a .do script does not produce an output file (eg.
all.do, clean.do), it also does not show up in this list.
The output of redo-targets might be useful in a
semi-automated `clean.do` target; you could delete all the
known targets, thus forcing them to be rebuilt next time.
Each filename is on a separate line. The filenames are not
guaranteed to be in any particular order.
All filenames are printed relative the current directory.
The list is not filtered in any way; it contains *all* the
target filenames from the entire project. Remember that
the redo database may span more than just your project, so
you might need to filter the list before using it. (A
useful heuristic might be to remove any line starting with
'../' since it often refers to a target you don't care
about.)
If you want a list of only out-of-date targets, use
`redo-ood`(1). If you want a list of sources (dependencies
that aren't targets), use `redo-sources`(1).
# REDO
Part of the `redo`(1) suite.
# CREDITS
The original concept for `redo` was created by D. J.
Bernstein and documented on his web site
(http://cr.yp.to/redo.html). This independent implementation
was created by Avery Pennarun and you can find its source
code at http://github.com/apenwarr/redo.
# SEE ALSO
`redo`(1), `redo-ood`(1), `redo-sources`(1)

104
docs/redo-whichdo.md Normal file
View file

@ -0,0 +1,104 @@
# NAME
redo-whichdo - show redo's search path for a .do file
# SYNOPSIS
redo-whichdo &lt;target>
# DESCRIPTION
`redo`(1) and `redo-ifchange`(1) build their targets by executing a ".do
file" script with appropriate arguments. .do files are searched starting
from the directory containing the target, and if not found there, up the
directory tree until a match is found.
To help debugging your scripts when redo is using an unexpected .do file, or
to write advanced scripts that "proxy" from one .do file to another, you
can use `redo-whichdo` to see the exact search path that `redo` uses.
The output format lists potential .do files, one per line, in order of
preference, separated by newline characters, and stopping once a
matching .do file has been found. If the return code is zero,
the last line is a .do file that actually exists; otherwise the entire
search path has been exhausted (and printed).
# EXAMPLE
Here's a typical search path for a source file (`x/y/a.b.o`). Because the
filename contains two dots (.), at each level of the hierarchy, `redo` needs
to search `default.b.o.do`, `default.o.do`, and `default.do`.
```sh
$ redo-whichdo x/y/a.b.o; echo $?
x/y/a.b.o.do
x/y/default.b.o.do
x/y/default.o.do
x/y/default.do
x/default.b.o.do
x/default.o.do
x/default.do
default.b.o.do
default.o.do
0
```
You might use `redo-whichdo` to delegate from one .do script to another,
using code like the following. This gets a little tricky because not only
are you finding a new .do file, but you have `cd` to the .do file
directory and adjust `$1` and `$2` appropriately.
ofile=$PWD/$3
x1=$1
cd "$SRCDIR"
redo-whichdo "$x1" | {
ifcreate=
while read dopath; do
if [ ! -e "$dopath" ]; then
ifcreate="$ifcreate $dopath"
else
redo-ifcreate $ifcreate
redo-ifchange "$dopath"
dofile=${dopath##*/}
dodir=${dopath%$dofile}
# Create updated $1 and $2 for the new .do file
x1_rel=${x1#$dodir}
ext=${dofile##*default}
if [ "$ext" != "$dofile" ]; then
ext=${ext%.do}
else
ext=''
fi
x2_rel=${x1#$dodir}
x2_rel=${x2_rel%$ext}
cd "$dodir"
set -- "$x1_rel" "$x2_rel" "$ofile"
. "./$dofile"
exit
fi
done
exit 3
}
# REDO
Part of the `redo`(1) suite.
# CREDITS
The original concept for `redo` was created by D. J.
Bernstein and documented on his web site
(http://cr.yp.to/redo.html). This independent implementation
was created by Avery Pennarun and you can find its source
code at http://github.com/apenwarr/redo.
# SEE ALSO
`redo`(1), `redo-ifchange`(1), `redo-ifcreate`(1)

290
docs/redo.md Normal file
View file

@ -0,0 +1,290 @@
# NAME
redo - rebuild target files when source files have changed
# SYNOPSIS
redo [options...] [targets...]
# DESCRIPTION
redo is a simple yet powerful tool for rebuilding target
files, and any of their dependencies, based on a set of
rules. The rules are encoded in simple `sh`(1) scripts
called '.do scripts.'
redo supports GNU `make`(1)-style parallel builds using the
`-j` option; in fact, redo's parallel jobserver is compatible
with GNU Make, so redo and make can share build tokens with
each other. redo can call a sub-make (eg. to build a
subproject that uses Makefiles) or vice versa (eg. if a
make-based project needs to build a redo-based subproject).
Unlike make, redo does not have any special syntax of its
own; each *target* is built by running a .do file, which is
simply a shell script that redo executes for you with a
particular environment and command-line arguments.
If no *targets* are specified, redo pretends you specified
exactly one target named `all`.
Note that redo *always* rebuilds the given targets
(although it may skip rebuilding the targets' dependencies
if they are up to date). If you only want to rebuild
targets that are not up to date, use `redo-ifchange`(1)
instead.
A .do script can call redo recursively to build its
dependencies.
To avoid confusion caused by multiple programs trying to
use the terminal, inside .do scripts, stdin is normally
redirected to /dev/null. The only exception is if the `-j`
option is *not* given and `--no-log` is used.
# OPTIONS
-j, --jobs=*maxjobs*
: execute at most *maxjobs* .do scripts in parallel. The
default value is 1.
-d, --debug
: print dependency checks as they happen. You can use
this to figure out why a particular target is/isn't being
rebuilt when your .do script calls it using
`redo-ifchange`.
-v, --verbose
: pass the -v option to /bin/sh when executing scripts.
This normally causes the shell to echo the .do script lines
to stderr as it reads them. Most shells will print the
exact source line (eg. `echo $3`) and not the
substituted value of variables (eg. `echo
mytarget.redo.tmp`).
-x, --xtrace
: pass the -x option to /bin/sh when executing scripts.
This normally causes the shell to echo exactly which
commands are being executed. Most shells will print
the substituted variables (eg. `echo
mytarget.redo.tmp`) and not the original source line
(eg. `echo $3`).
-k, --keep-going
: keep building as many targets as possible even if some
of them return an error. If one target fails, any
target that depends on it also cannot be built, of course.
--shuffle
: randomize the order in which requested targets are
built. Normally, if you run `redo a b c`, the targets
will be built exactly in that order: first `a`, then
`b`, then `c`. But if you use `-j`, they might end up
being built in parallel, so it isn't safe to rely on
this precise ordering. Using `--shuffle`, redo will
build its targets in random order even without `-j`,
which makes it easier to find accidental dependency
problems of this sort. NOTE: if you really just want
to guarantee that `a` is built, then `b`, then `c`, you
can just run three `redo` commands consecutively.
Because your .do script is just a script, it will not
be accidentally parallelized.
--no-details
: display *only* the messages from redo itself, not the other messages
produced by build scripts. Generally this gives you a list of which
targets were built, but not detailed logs, warnings, or errors.
--no-status
: don't display the running build status at the bottom of the screen.
(Unless this option is specified, the status line will be enabled
if using --follow, if stderr is a terminal.) If stderr is *not* a
terminal, you can force enable the status line using --status.
--no-pretty
: display "raw" redo log lines (@@REDO events) rather than using a
human-readable format. The default is --pretty.
--no-color
: when using --pretty and writing to a terminal, colorize the output to
make results stand out more clearly. If not writing to a terminal, you
can use --color to force colorized output.
--no-log
: don't capture stderr log messages from build scripts. This prevents
redo-log from redisplaying the logs later, and if using --jobs, causes
output from all parallel jobs to be jumbled together. This was the
only behaviour available before redo-0.30. The default is --log.
--debug-locks
: print messages about acquiring, releasing, and waiting
on locks. Because redo can be highly parallelized,
one instance may end up waiting for a target to be
built by some other instance before it can continue.
If you suspect this is causing troubles, use this
option to see which instance is waiting and when.
--debug-pids
: add the process id of the particular redo instance to each
output message. This makes it easier to figure out
which sub-instance of redo is doing what.
--version
: displays the redo version number.
# DISCUSSION
The core of redo is extremely simple. When you type `redo
targetname`, then it will search for a matching .do file
based on a simple algorithm. For example, given a target
named `mytarget.a.b.c.d`, redo will look for a .do file in
the following order:
- mytarget.a.b.c.d.do
- default.a.b.c.d.do
- default.b.c.d.do
- default.c.d.do
- default.d.do
- default.do
In all cases, the .do file must be in the same directory as
the target file, or in one of the target's parent
directories. For example, if given a target named
`../a/b/xtarget.y`, redo will look for a .do file in the
following order:
- $PWD/../a/b/xtarget.y.do
- $PWD/../a/b/default.y.do
- $PWD/../a/b/default.do
- $PWD/../a/default.y.do
- $PWD/../a/default.do
- $PWD/../default.y.do
- $PWD/../default.do
The first matching .do file is executed as a `/bin/sh`
script. The .do script is always executed with the current
working directory set to the directory containing the .do
file. Because of that rule, the
following two commands always have exactly identical
behaviour:
redo path/to/target
cd path/to && redo target
(Note: in `make`(1), these commands have confusingly
different semantics. The first command would look for a
target named `path/to/target` in `./Makefile`, while the
second command would look for a target named `target` in
`./path/to/Makefile`. The two Makefiles might give
completely different results, and it's likely that the
first command would have incomplete dependency information.
redo does not have this problem.)
The three arguments passed to the .do script are:
- $1: the target name (eg. mytarget.a.b)
- $2: the basename of the target, minus its extension (eg. mytarget)
- $3: a temporary filename that the .do script should write
its output to.
Instead of using $3, the .do script may also write the
produced data to stdout.
If the .do file is in the same directory as the target, $1
is guaranteed to be a simple filename (with no path
component). If the .do file is in a parent directory of
the target, $1 and $3 will be relative paths (ie. will
contain slashes).
redo is designed to update its targets atomically, and only
if the do script succeeds (ie. returns a zero exit code).
Thus, you should never write directly to the target file,
only to $3 or stdout.
Normally, a .do script will call other .do scripts
recursively, by running either `redo` (which will always
build the sub-target) or `redo-ifchange` (which only
rebuilds the sub-target if its dependencies have changed).
Running `redo-ifchange` is also the way your .do script
declares dependencies on other targets; any target that is
`redo-ifchange`d during your .do script's execution is both
executed (if needed) and added as a dependency.
You may have heard that 'recursive make is considered
harmful' (http://miller.emu.id.au/pmiller/books/rmch/).
Unlike `make`(1), redo does correct locking, state
management, and global dependency checking, so none of the
arguments in that essay apply to redo. In fact, recursive
redo is really the only kind of redo.
# RELATED COMMANDS
When writing a .do script, it will probably need to run
one or more of the following commands:
`redo [targets...]`
: to build a sub-target unconditionally.
`redo-ifchange [targets...]`
: to declare dependencies on the given files, and build them if
they don't yet exist or are outdated.
`redo-ifcreate [sources...]`
: to tell redo that the current target must be rebuilt if
the given source files (which must not yet exist) get created.
`redo-always`
: to tell redo that the current target must always be
rebuilt, even if none of its dependencies have changed.
(You might need this for targets that depend on more than just
file contents. For example, the output of `ls *.c`
changes when files are created or deleted, but there is no way
for redo to know that without re-running the command.)
`echo "stamp contents..." | redo-stamp`
: to tell redo that even though the current target has
been rebuilt, it may not actually be any different from
the previous version, so targets that depend on it
might not need to be rebuilt. Often used in
conjunction with `redo-always` to reduce the impact of
always rebuilding a target.
There are also some less common ones:
`redo-ood`
: Get a list of all known targets that have been built before, but
are currently out of date.
`redo-targets`
: Get a list of all known targets that have been built before.
`redo-sources`
: Get a list of all known redo source files that still exist.
`redo-whichdo <target>`
: Explain the search path used to find a .do file for the given
target.
# CREDITS
The original concept for `redo` was created by D. J.
Bernstein and documented on his web site
(http://cr.yp.to/redo.html). This independent implementation
was created by Avery Pennarun and you can find its source
code at http://github.com/apenwarr/redo.
# SEE ALSO
`sh`(1), `make`(1),
`redo-ifchange`(1), `redo-ifcreate`(1), `redo-always`(1),
`redo-stamp`(1), `redo-ood`(1), `redo-targets`(1), `redo-sources`(1),
`redo-whichdo`(1)

87
docs/t/testitem.md Normal file
View file

@ -0,0 +1,87 @@
% redo(1) Redo %VERSION%
% Avery Pennarun <apenwarr@gmail.com>
% %DATE%
# NAME
redo-always - mark the current target as always needing to be rebuilt
# SYNOPSIS
redo-always
# DESCRIPTION
Normally redo-always is run from a .do file that has been
executed by `redo`(1). See `redo`(1) for more details.
This is a "quoted string."
"entirely quoted"
.starting with dot
I'm a \big \\backslasher!
This **line** has _multiple_ `formats`(interspersed) *with* each _other_
and *here is a multi
line italic.*
This line has an & and a < and a >.
- "line starting with quoted"
Here's some code
with indentation
yay! (a \backslash and <brackets>)
"foo"
skipped a line
indented
another skip
- -starting with dash
- .starting with dot
chicken
- list item
with more text
and even more
- second list
- third list
wicken
- list 1a
- list 1b
- list 2
- list 3
barf
First line
: definition list.
with
multiple
lines!
--item=*value*
: a description.
`-x`
: more stuff. if you had a lot of text, this is what it
would look like. It goes on and on and on.
a line with *altogether* "too much" stuff on it to realistically *make* it in a definition list
: and yet, here we are.
# SEE ALSO
`redo`(1), `redo-ifcreate`(1), `redo-ifchange`(1), `redo-stamp`(1)

4
docs/test.do Normal file
View file

@ -0,0 +1,4 @@
# We don't normally build the cookbook examples, because they're
# not really part of redo itself. But build them when testing,
# as a basic check that redo (and the examples) are valid.
redo cookbook/all