forks/apenwarr-redo
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Directory reorg: move code into redo/, generate binaries in bin/.

Browse source 
It's time to start preparing for a version of redo that doesn't work
unless we build it first (because it will rely on C modules, and
eventually be rewritten in C altogether).

To get rolling, remove the old-style symlinks to the main programs, and
rename those programs from redo-*.py to redo/cmd_*.py.  We'll also move
all library functions into the redo/ dir, which is a more python-style
naming convention.

Previously, install.do was generating wrappers for installing in
/usr/bin, which extend sys.path and then import+run the right file.
This made "installed" redo work quite differently from running redo
inside its source tree.  Instead, let's always generate the wrappers in
bin/, and not make anything executable except those wrappers.

Since we're generating wrappers anyway, let's actually auto-detect the
right version of python for the running system; distros can't seem to
agree on what to call their python2 binaries (sigh). We'll fill in the
right #! shebang lines.  Since we're doing that, we can stop using
/usr/bin/env, which will a) make things slightly faster, and b) let us
use "python -S", which tells python not to load a bunch of extra crap
we're not using, thus improving startup times.

Annoyingly, we now have to build redo using minimal/do, then run the
tests using bin/redo.  To make this less annoying, we add a toplevel
./do script that knows the right steps, and a Makefile (whee!) for
people who are used to typing 'make' and 'make test' and 'make clean'.
This commit is contained in:
Avery Pennarun 2018-12-03 21:39:15 -05:00
commit f6fe00db5c
140 changed files with 256 additions and 99 deletions
Download patch file Download diff file Expand all files Collapse all files

322
Documentation/cookbook/defaults/index.md
View file

@ -1,322 +0,0 @@
### Building a text preprocessor
Let's say we want to process a bunch of files, each in the same way. An
easy example is a text preprocessor: we'll look through the text and replace
variables of the form %VARNAME% with real content. In our example, we'll
use variable %VERSION%, representing a version number, and %DATE%, with
today's date.
To play with this code on your own machine, get the [redo
source code](https://github.com/apenwarr/redo) and look in the
`Documentation/cookbook/defaults/` directory.
### Input files
Let's create some input files we can use. The input to our
preprocessor will have file extension `.in`. The output can be in various
formats.
<pre><code src='test.txt.in'></code></pre>
<pre><code lang='c' src='include/version.h.in'></code></pre>
<pre><code lang='python' src='version.py.in'></code></pre>
For fun, let's also make a python test program that calls the newly
generated version.py:
<pre><code lang='python' src='test.py'></code></pre>
Finally, we need to provide values for the variables. Let's put each one
in its own file, named `version` and `date`, respectively.
<pre><b style='text-align: center; display: block'>version</b
><code>1.0</code></pre>
<pre><b style='text-align: center; display: block'>date</b
><code>1970-01-01</code></pre>
### default.do files
Now we want to teach redo how to do our substitutions: how to, in general,
generate file X from file X.in.
We could write a separate .do file for every output file X. For example, we
might make `test.txt.do` and `version.py.do`. But that gets tedious. To
make it easier, if there is no specific `X.do` for a target named X, redo
will try using `default.do` instead. Let's write a default.do for our text
preprocessor.
<pre><code lang='sh' src='default.do'></code></pre>
If `default.do` is asked to build `X`, and there exists a file named `X.in`, we
use `sed` to do our variable substitutions. In that case, `default.do` uses
redo-ifchange to depend on `X.in`, `version`, and `date`. If a file named
`X.in` does *not* exist, then we don't know what to do, so we give an error.
On the other hand, if we try to generate a file that *already* exists, like
`test.py`, redo does not call default.do at all. redo only tries to create
files that don't exist, or that were previously generated by redo. This
stops redo from accidentally overwriting your work.
```shell
$ redo test.txt
redo test.txt
$ cat test.txt
This is the documentation for MyProgram version
1.0. It was generated on 1970-01-01.
$ redo chicken
redo chicken
default.do: Fatal: don't know how to build 'chicken'
redo chicken (exit 99)
$ redo version.py
redo version.py
# test.py was created by us, so it's a "source" file.
# redo does *not* call default.do to replace it.
$ redo test.py
redo: test.py: exists and not marked as generated; not redoing.
$ python test.py
Version '1.0' has build date '1970-01-01'
```
Nice!
While we're here, let's make an `all.do` so that we don't have to tell redo
exactly which files to rebuild, every single time.
<pre><code lang='sh' src='all.do'></code></pre>
Results:
```shell
$ redo
redo all
redo test.txt
redo version.py
redo include/version.h
# input files didn't change, so nothing to rebuild
$ redo
redo all
$ touch test.txt.in
$ redo
redo all
redo test.txt
```
### Auto-generating the version and date (redo-always and redo-stamp)
Of course, in a real project, we won't want to hardcode the version number
and date into a file. Ideally, we can get the version number from a version
control system, like git, and we can use today's date.
To make that happen, we can replace the static `version` and `date` files with
`version.do` and `date.do`. `default.do` already uses redo-ifchange to
depend on `version` and `date`, so redo will create them as needed, and
if they change, redo will rebuild all the targets that depend on them.
However, the `version` and `date` files are special: they depend on the
environment outside redo itself. That is, there's no way to declare a
dependency on the current date. We might generate the `date` file once, but
tomorrow, there's no way for redo to know that its value should change.
To handle this situation, redo has the `redo-always` command. If we run
redo-always from a .do file, it means every time someone depends on that
target, it will be considered out-of-date and need to be rebuilt. The
result looks like this:
```shell
$ redo
redo all
redo test.txt
redo version
redo date
redo version.py
redo include/version.h
# version.do and date.do are redo-always, so
# everything depending on them needs to rebuild
# every time.
$ redo
redo all
redo test.txt
redo version
redo date
redo version.py
redo include/version.h
```
Of course, for many uses, that's overcompensating: the version number and
date don't change *that* often, so we might end up doing a lot of
unnecessary work on every build. To solve that, there's `redo-stamp`.
redo-stamp does the opposite of redo-always: while redo-always makes things
build *more* often, redo-stamp makes things build *less* often.
Specifically, it lets a .do file provide a "stamp value" for its output; if
that stamp value is the same as before, then the target should be considered
unchanged after all.
The most common stamp value is just the content itself. Since in redo, we
write the content to $3, we can also read it back from $3:
<pre><code lang='sh' src='version.do'></code></pre>
<pre><code lang='sh' src='date.do'></code></pre>
And the final result is what we want. Although `version` and `date` are
generated every time, the targets which depend on them are not:
```shell
$ redo clean
redo clean
# version and date are generated just once per run,
# the first time they are used.
$ redo
redo all
redo test.txt
redo version
redo date
redo version.py
redo include/version.h
# Here, (test.txt) means redo is considering building
# test.txt, but can't decide yet. In order to decide,
# it needs to first build date and version. After
# that, it decides not to build test.txt after all.
$ redo
redo all
redo (test.txt)
redo date
redo version
```
### Temporary overrides
Sometimes you want to override a file even if it *is* a target (ie. it has
previously been built by redo and has a valid .do file associated with it).
In our example, maybe you want to hardcode the version number because you're
building a release. This is easy: redo notices whenever you overwrite a
file from outside redo, and will avoid replacing that file until you
subsequently delete it:
```shell
$ echo "1.0" >version
$ redo
redo all
redo (test.txt)
redo date
redo: version - you modified it; skipping
redo test.txt
redo version.py
redo include/version.h
$ redo
redo all
redo (test.txt)
redo date
redo: version - you modified it; skipping
$ rm version
$ redo
redo all
redo (test.txt)
redo date
redo version
redo test.txt
redo version.py
redo include/version.h
```
### default.do, subdirectories, and redo-whichdo
There's one more thing we should mention, which is the interaction of
default.do with files in subdirectories. Notice that we are building
`include/version.h` in our example:
```shell
$ redo include/version.h
redo include/version.h
redo version
redo date
$ cat include/version.h
// C/C++ header file identifying the current version
#ifndef __VERSION_H
#define __VERSION_H
#define VERSION "redo-0.31-3-g974eb9f"
#define DATE "2018-11-26"
#endif // __VERSION_H
```
redo works differently from the `make` command when you ask it to build
files in subdirectories. In make's case, it always looks for a `Makefile`
in the *current* directory, and uses that for all build instructions. So
`make include/version.h` and `cd include && make version.h` are two
different things; the first uses `Makefile`, and the second uses
`include/Makefile` (or crashes if the latter does not exist).
redo, on the other hand, always uses the same formula to find a .do file for
a particular target. For a file named X, that formula is as follows:
- first, try X.do
- then try default.do
- then try ../default.do
- then try ../../default.do
- ...and so on...
(Note: for targets with an extension, like X.o, redo actually tries even
more .do files, like `default.o.do` and `../default.o.do`. For precise
details, read the [redo man page](../../redo).)
You can see which .do files redo considers for a given target by using the
`redo-whichdo` command. If redo-whichdo returns successfully, the last name
in the list is the .do file it finally decided to use.
```shell
$ redo-whichdo include/version.h
include/version.h.do
include/default.h.do
include/default.do
default.h.do
default.do
```
### Redo always runs in the .do file's directory
To ensure consistency, redo always changes the current directory to
the directory *containing the selected .do file* (**not** the directory
containing the target, if they are different). As a result,
`redo include/version.h` and `cd include && redo version.h` always have
exactly the same effect:
```shell
$ redo include/version.h
redo include/version.h
redo version
redo date
$ (cd include && redo version.h)
redo version.h
redo ../version
redo ../date
```
(redo's display is slightly different between the two: it always shows the
files it's building relative to the $PWD at the time you started redo.)
This feature is critical to redo's recursive nature; it's the reason that
essays like [Recursive Make Considered Harmful](http://aegis.sourceforge.net/auug97.pdf)
don't apply to redo. Any redo target, anywhere in your source tree, can
use redo-ifchange to depend on any of your other targets, and the dependency
will work right.
Why does redo change to the directory containing the .do file, instead of
the directory containing the target? Because usually, the .do file needs to
refer to other dependencies, and it's easier to always express those
dependencies without adjusting any paths. In our text preprocessor example,
`default.do` does `redo-ifchange version date`; this wouldn't work properly
if it were running from the `include/` directory, because there are no files
named `version` and `date` in there.
Similarly, when compiling C programs, there are
usually compiler options like `-I../mylib/include`. If we're compiling
`foo.o` and `mydir/bar.o`, we would like `-I../mylib/include` to have the
same meaning in both cases.