We believe that, unlike most programs, it's actually possible to "finish" redo, in the sense of eventually not needing to extend its semantics or add new features. That's because redo is a pretty low-level system that just provides some specific features (dependency checking, parallelism, log linearization, inter-process locking). It's the job of your build scripts to tie those features together in the way you want. `make` has its own imperative syntax, which creates a temptation to add new built-in functions and syntax extensions. In more "declarative" build systems, there's a constant need to write new extension modules or features in order to create functionality that wasn't available declaratively. redo avoids that by using a turing-complete language to run your builds. You should be able to build anything at all with redo, just by writing your .do scripts the way you want. Thus, the only things that need to be added to redo (other than portability and bug fixes, which will likely be needed forever) are to fix gaps in redo's model that prevent you from getting your work done. This document describes the gaps we're currently aware of. Note that all of the items in this document are still unimplemented. In most cases, that's because we haven't yet settled on a final design, and it's still open to discussion. The place to discuss design issues is the [mailing list](../Contributing/#mailing-list). ### default.do search path, and separated build directories One of the most controversial topics in redo development, and for developers trying to use redo, is: where do you put all those .do scripts? redo searches hierarchically up the directory tree from a target's filename, hoping to find a `default*.do` file that will match, and then uses the first one it finds. This method is rather elegant when it works. But many developers would like to put their output files into a separate directory from their source files, and that output directory might not be a subdirectory of the main project (for example, if the main project is on a read-only filesystem). There are already a few ways to make this work, such as placing a single `default.do` "proxy" or "delegation" script at the root of the output directory, which will bounce requests to .do files it finds elsewhere. One nice thing about this feature is it doesn't require any changes to redo itself; redo already knows how to call your toplevel default.do script. However, some people find the delegation script to be inelegant and complicated. Other options include searching inside a known subdirectory name (eg. `do/`), which could be a symlink; or adding a `.dopath` file which tells redo to search elsewhere. So far, we haven't settled on the best design, and discussion is welcome. In the meantime, you can write a delegation script (TODO: link to example) for your project. Because this requires no special redo features, it's unlikely to break in some later version of redo, even if we add a new method. ### .do files that produce directories Sometimes you want a .do file to produce multiple output files in a single step. One example is an autoconf `./configure` script, which might produce multiple files. Or, for example, look at the [LaTeX typesetting example](../cookbook/latex/) in the redo cookbook. In the purest case, generating multiple outputs from a single .do file execution violates the redo semantics. The design of redo calls for generating *one* output file from *zero or more* input files. And most of the time, that works fine. But sometimes it's not enough. Currently (like in the LaTeX example linked above) we need to resolve this problem by taking advantage of "side effects" in redo: creating a set of files that are unknown to redo, but sit alongside the "known" files in the filesystem. But this has the annoying limitation that you cannot redo-ifchange directly on the file you want, if it was generated this way. For example, if `runconfig.do` generates `Makefile` and `config.h`, you must not `redo-ifchange config.h` directly; there is no .do file for `config.h`. You must `redo-ifchange runconfig` and then *use* `config.h`. (There are workarounds for that workaround: for example, `runconfig.do` could put all its output files in a `config/` directory, and then you could have a `config.h.do` that does `redo-ifchange runconfig` and `cp config/config.h $3`. Then other scripts can `redo-ifchange config.h` without knowing any more about it. But this method gets tedious.) One suggestion for improving the situation would be to teach redo about "directory" targets. For example, maybe we have a `config.dir.do` that runs `./configure` and produces files in a directory named `config`. The `.dir.do` magic suffix tells redo that if someone asks for `config/config.h`, it must first try to instantiate the directory named `config` (using `config.dir.do`), and only then try to depend on the file inside that directory. There are a number of holes in this design, however. Notably, it's not obvious how redo should detect when to activate the magic directory feature. It's easy when there is a file named `config.dir.do`, but much less obvious for a file like `default.dir.do` that can construct certain directory types, but it's not advertised which ones. This particular cure may turn out to be worse than the disease. ### Per-target-directory .redo database An unexpectedly very useful feature of redo is the ability to "redo from anywhere" and get the same results: ```shell $ cd /a/b/c $ redo /x/y/z/all ``` should have the same results as ```shell $ cd /x/y/z $ redo all ``` Inside a single project, this already works. But as redo gets used more widely, and in particular when you have multiple redo-using projects that want to refer to other redo-using projects, redo can get confused about where to put its `.redo` state database. Normally, it goes into a directory called `$REDO_BASE`, the root directory of your project. But if a .do script refers to a directory outside or beside the root, this method doesn't work, and redo gets the wrong file state information. Further complications arise in the case of symlinks. For example, if you ask redo to build `x/y/z/file` but `y` is a symlink to `q`, then redo will effectively end up replacing `x/q/z/file` when it replces `x/y/z/file`, since they're the same. If someone then does `redo-ifchange x/q/z/file`, redo may become confused about why that file has "unexpectedly" changed. The fix for both problems is simple: put one `.redo` database in every directory that contains target files. The `.redo` in each directory contains information only about the targets in that directory. As a result, `x/y/z/file` and `x/q/z/file` will share the same state database, `x/q/z/.redo`, and building either target will update the state database's understanding of the file called `file` in the same directory, and there will be no confusion. Similarly, one redo-using project can refer to targets in another redo-using project with no problem, because redo will no longer have the concept of a `$REDO_BASE`, so there is no way to talk about targets "outside" the `$REDO_BASE`. Note that there is no reason to maintain a `.redo` state database in *source* directories (which might be read-only), only target directories. This is because we store `stat(2)` information for each dependency anyway, so it's harmless if multiple source filenames are aliases for the same underlying content. ### redo-{sources,targets,ood} should take a list of targets With the above change to a per-target-directory `.redo` database, the original concept of the `redo-sources`, `redo-targets`, and `redo-ood` commands needs to change. Currently they're defined to list "all" the sources, targets, and out-of-date targets, respectively. But when there is no single database reflecting the entire filesystem, the concept of "all" becomes fuzzy. We'll have to change these programs to refer to "all (recursive) dependencies of targets in the current directory" by default, or of all targets listed on the command line otherwise. This is probably more useful than the current behaviour anyway, since in a large project, one rarely wants to see a complete list of all sources and targets. ### Deprecating "stdout capture" behaviour The [original design for redo](http://cr.yp.to/redo.html) specified that a .do script could produce its output either by writing to stdout, or by writing to the file named by the `$3` variable. Experience has shown that most developers find this very confusing. In particular, results are undefined if you write to *both* stdout and `$3`. Also, many programs (including `make`!) write their log messages to stdout when they should write to stderr, so many .do scripts need to start with `exec >&2` to avoid confusion. In retrospect, automatically capturing stdout was probably a bad idea. .do scripts should intentionally redirect to `$3`. To enforce this, we could have redo report an error whenever a .do script returns after writing to its stdout. For backward compatibility, we could provide a command-line option to downgrade the error to a warning. ### Deprecating environment variable sharing In redo, it's considered a poor practice to pass environment variables (and other process attributes, like namespaces) from one .do script to another. This is because running `redo-ifchange /path/to/file` should always run `file`'s .do script with exactly the same settings, whether you do it from the toplevel from from deep inside a tree of dependencies. If an environment variable set in one .do script can change what's seen by an inner .do script, this breaks the dependency mechanism and makes builds less repeatable. To make it harder to do this by accident, redo could intentionally wipe all but a short whitelist of allowed environment variables before running any .do script. As a bonus, by never sharing any state outside the filesystem, it becomes much more possible to make a "distributed redo" that builds different targets on different physical computers. ### redo-recheck command Normally, redo only checks any given file dependency at most once per session, in order to reduce the number of system calls executed, thus greatly speeding up incremental builds. As a result, `redo-ifchange` of the same target will only execute the relevant .do script at most once per session. In some situations, notably integration tests, we want to force redo to re-check more often. Right now there's a hacky script called `t/flush-cache` in the redo distribution which does this, but it relies on specific knowledge of the .redo directory's database format, which means it only works in this specific version of redo; this prevents the integration tests from running (and thus checking compatibility with) competing redo implementations. If we standardized a `redo-recheck` command, which would flush the cache for the targets given on the command line, and all of their dependencies, this sort of integration test could work across multiple redo versions. For redo versions which don't bother caching, `redo-recheck` could be a null operation. ### tty input Right now, redo only allows a .do file to request input from the user's terminal if using `--no-log` and *not* using the `-j` option. Terminal input is occasionally useful for `make config` interfaces, but parallelism and log linearization make the console too cluttered for a UI to work. The ninja build system has a [console pool](https://ninja-build.org/manual.html#_the_literal_console_literal_pool) that can contain up to one job at a time. When a job is in the console pool, it takes over the console entirely. We could probably implement something similar in redo by using POSIX job control features, which suspend subtasks whenever they try to read from the tty. If we caught the suspension signal and acquired a lock, we could serialize console access. Whether the complexity of this feature is worthwhile is unclear. Maybe it makes more sense just to have a './configure' script that runs outside the redo environment, but still can call into redo-ifchange if needed. ### redo-lock command Because it supports parallelism via recursion, redo automatically handles inter-process locking so that only one instance of redo can try to build a given target at a time. This sort of locking turns out to be very useful, but there are a few situations where requiring redo to "build a target by calling a .do file" in order to acquire a lock becomes awkward. For example, imagine redo is being used to call into `make` to run arbitrary `Makefile` targets. `default.make.do` might look like this: ```sh make "$2" ``` redo will automatically prevent two copies of `redo all.make` from running at once. However, if someone runs `redo all.make myprogram.make`, then two copies of `make` will execute at once. This *might* be harmless, but if the `all` target in the `Makefile` has a dependency on `myprogram`, then we will actually end up implicitly building `myprogram` from two places at once: from the `myprogram` part of `all.make` and from `myprogram.make`. In hairy situations like that, it would be nice to serialize all access inside `default.make.do`, perhaps like this: ```sh redo-lock make.lock make "$2" ``` This would create a redo-style lock on the (virtual) file `make.lock`, but then instead of trying to `redo make.lock`, it would run the given command, in this case `make "$2"`. It's unclear whether this feature is really a good idea. There are other (convoluted) ways to achieve the same goal. Nevertheless, it would be easy enough to implement. And redo versions that don't support parallelism could just make redo-lock a no-op, since they guarantee serialization in all cases anyway. ### Include a (minimal) POSIX shell A common problem when writing build scripts, both in `make` and in redo, is gratuitous incompatibility between all the available POSIX-like unix shells. Nowadays, most shells support [various pure POSIX sh features](https://apenwarr.ca/log/20110228), but there are always glitches. In some cases, POSIX doesn't define the expected behaviour for certain situations. In others, shells like `bash` try to "improve" things by changing the syntax in non-POSIX ways. Or maybe they just add new backward-compatible features, which you then rely on accidentally because you only tested your scripts with `bash`. redo on Windows using something like [MSYS](http://www.mingw.org/wiki/msys) is especially limited by the lack of (and oddity of) available unix tools. To avoid all these portability problems for .do script maintainers, we might consider bundling redo with a particular (optional) sh implementation, and maybe also unix-like tools, that it will use by default. An obvious candidate would be busybox, which has a win32 version called [busybox-w32](https://frippery.org/busybox/). ### redoconf redo is fundamentally a low-level tool that doesn't know as much about compiling specific programming languages as do higher-level tools like [cmake](https://cmake.org/). Similarly, `make` doesn't know much about specific programming languages (and what it does know is hopelessly out of date, but cannot be deleted or updated because it would break backward compatibility with old Makefiles). This is why `autoconf` and `automake` were created: to automatically fill in the language- and platform-specific blanks, while letting `make` still handle executing the low level instructions. It might be useful to have a redo-native autoconf/automake-like system, although you can already use autoconf with redo, so this might not be essential.