From 280cf6f3aa10d1afb06a418b48f07d222a0f7504 Mon Sep 17 00:00:00 2001 From: Avery Pennarun Date: Mon, 3 Dec 2018 01:45:36 -0500 Subject: [PATCH] Add Roadmap.md. --- Documentation/Roadmap.md | 329 +++++++++++++++++++++++++++++++++++++++ mkdocs.yml | 1 + 2 files changed, 330 insertions(+) create mode 100644 Documentation/Roadmap.md diff --git a/Documentation/Roadmap.md b/Documentation/Roadmap.md new file mode 100644 index 0000000..9c7666a --- /dev/null +++ b/Documentation/Roadmap.md @@ -0,0 +1,329 @@ +We believe that, unlike most programs, it's actually possible to "finish" +redo, in the sense of eventually not needing to extend its semantics or add +new features. That's because redo is a pretty low-level system that just +provides some specific features (dependency checking, parallelism, log +linearization, inter-process locking). It's the job of your build scripts +to tie those features together in the way you want. + +`make` has its own imperative syntax, which creates a temptation to add +new built-in functions and syntax extensions. In more +"declarative" build systems, there's a constant need to write new extension +modules or features in order to create functionality that wasn't available +declaratively. redo avoids that by using a turing-complete language +to run your builds. You should be able to build anything at all with redo, +just by writing your .do scripts the way you want. + +Thus, the only things that need to be added to redo (other than portability +and bug fixes, which will likely be needed forever) are to fix gaps in +redo's model that prevent you from getting your work done. This document +describes the gaps we're currently aware of. + +Note that all of the items in this document are still unimplemented. In +most cases, that's because we haven't yet settled on a final design, and +it's still open to discussion. The place to discuss design issues is the +[mailing list](../Contributing/#mailing-list). + + +### default.do search path, and separated build directories + +One of the most controversial topics in redo development, and for developers +trying to use redo, is: where do you put all those .do scripts? + +redo searches hierarchically up the directory tree from a target's filename, +hoping to find a `default*.do` file that will match, and then uses the first +one it finds. This method is rather elegant when it works. But many +developers would like to put their output files into a separate directory +from their source files, and that output directory might not be a +subdirectory of the main project (for example, if the main project is on a +read-only filesystem). + +There are already a few ways to make this work, such as placing a single +`default.do` "proxy" or "delegation" script at the root of the output +directory, which will bounce requests to .do files it finds elsewhere. One +nice thing about this feature is it doesn't require any changes to redo +itself; redo already knows how to call your toplevel default.do script. +However, some people find the delegation script to be inelegant and +complicated. + +Other options include searching inside a known subdirectory name (eg. +`do/`), which could be a symlink; or adding a `.dopath` file which tells +redo to search elsewhere. + +So far, we haven't settled on the best design, and discussion is welcome. +In the meantime, you can write a delegation script (TODO: link to example) +for your project. Because this requires no special redo features, it's +unlikely to break in some later version of redo, even if we add a new +method. + + +### .do files that produce directories + +Sometimes you want a .do file to produce multiple output files in a single +step. One example is an autoconf `./configure` script, which might produce +multiple files. Or, for example, look at the [LaTeX typesetting +example](../cookbook/latex/) in the redo cookbook. + +In the purest case, generating multiple outputs from a single .do file +execution violates the redo semantics. The design of redo calls for +generating *one* output file from *zero or more* input files. And most of +the time, that works fine. But sometimes it's not enough. + +Currently (like in the LaTeX example linked above) we need to resolve this +problem by taking advantage of "side effects" in redo: creating a set of +files that are unknown to redo, but sit alongside the "known" files in the +filesystem. But this has the annoying limitation that you cannot +redo-ifchange directly on the file you want, if it was generated this way. +For example, if `runconfig.do` generates `Makefile` and `config.h`, you +must not `redo-ifchange config.h` directly; there is no .do file for +`config.h`. You must `redo-ifchange runconfig` and then *use* +`config.h`. + +(There are workarounds for that workaround: for example, `runconfig.do` +could put all its output files in a `config/` directory, and then you could +have a `config.h.do` that does `redo-ifchange runconfig` and `cp +config/config.h $3`. Then other scripts can `redo-ifchange config.h` +without knowing any more about it. But this method gets tedious.) + +One suggestion for improving the situation would be to teach redo about +"directory" targets. For example, maybe we have a `config.dir.do` that +runs `./configure` and produces files in a directory named `config`. The +`.dir.do` magic suffix tells redo that if someone asks for +`config/config.h`, it must first try to instantiate the directory named +`config` (using `config.dir.do`), and only then try to depend on the file +inside that directory. + +There are a number of holes in this design, however. Notably, it's not +obvious how redo should detect when to activate the magic directory feature. +It's easy when there is a file named `config.dir.do`, but much less obvious +for a file like `default.dir.do` that can construct certain directory types, +but it's not advertised which ones. + +This particular cure may turn out to be worse than the disease. + + +### Per-target-directory .redo database + +An unexpectedly very useful feature of redo is the ability to "redo from +anywhere" and get the same results: +```shell + $ cd /a/b/c + $ redo /x/y/z/all +``` +should have the same results as +```shell + $ cd /x/y/z + $ redo all +``` + +Inside a single project, this already works. But as redo gets used more +widely, and in particular when you have multiple redo-using projects that +want to refer to other redo-using projects, redo can get confused about +where to put its `.redo` state database. Normally, it goes into a directory +called `$REDO_BASE`, the root directory of your project. But if a .do +script refers to a directory outside or beside the root, this method doesn't +work, and redo gets the wrong file state information. + +Further complications arise in the case of symlinks. For example, if you +ask redo to build `x/y/z/file` but `y` is a symlink to `q`, then redo will +effectively end up replacing `x/q/z/file` when it replces `x/y/z/file`, +since they're the same. If someone then does `redo-ifchange x/q/z/file`, +redo may become confused about why that file has "unexpectedly" changed. + +The fix for both problems is simple: put one `.redo` database in every +directory that contains target files. The `.redo` in each directory +contains information only about the targets in that directory. As a result, +`x/y/z/file` and `x/q/z/file` will share the same state database, +`x/q/z/.redo`, and building either target will update the state database's +understanding of the file called `file` in the same directory, and there +will be no confusion. + +Similarly, one redo-using project can refer to targets in another redo-using +project with no problem, because redo will no longer have the concept of a +`$REDO_BASE`, so there is no way to talk about targets "outside" the +`$REDO_BASE`. + +Note that there is no reason to maintain a `.redo` state database in +*source* directories (which might be read-only), only target directories. +This is because we store `stat(2)` information for each dependency anyway, so +it's harmless if multiple source filenames are aliases for the same +underlying content. + + +### redo-{sources,targets,ood} should take a list of targets + +With the above change to a per-target-directory `.redo` database, the +original concept of the `redo-sources`, `redo-targets`, and `redo-ood` +commands needs to change. Currently they're defined to list "all" the +sources, targets, and out-of-date targets, respectively. But when there is +no single database reflecting the entire filesystem, the concept of "all" +becomes fuzzy. + +We'll have to change these programs to refer to "all (recursive) +dependencies of targets in the current directory" by default, or of all +targets listed on the command line otherwise. This is probably more useful +than the current behaviour anyway, since in a large project, one rarely +wants to see a complete list of all sources and targets. + + +### Deprecating "stdout capture" behaviour + +The [original design for redo](http://cr.yp.to/redo.html) specified that a +.do script could produce its output either by writing to stdout, or by +writing to the file named by the `$3` variable. + +Experience has shown that most developers find this very confusing. In +particular, results are undefined if you write to *both* stdout and `$3`. +Also, many programs (including `make`!) write their log messages to stdout +when they should write to stderr, so many .do scripts need to start with +`exec >&2` to avoid confusion. + +In retrospect, automatically capturing stdout was probably a bad idea. .do +scripts should intentionally redirect to `$3`. To enforce this, we could +have redo report an error whenever a .do script returns after writing to its +stdout. For backward compatibility, we could provide a command-line option +to downgrade the error to a warning. + + +### Deprecating environment variable sharing + +In redo, it's considered a poor practice to pass environment variables (and +other process attributes, like namespaces) from one .do script to another. +This is because running `redo-ifchange /path/to/file` should always run +`file`'s .do script with exactly the same settings, whether you do it from +the toplevel from from deep inside a tree of dependencies. If an +environment variable set in one .do script can change what's seen by an +inner .do script, this breaks the dependency mechanism and makes builds less +repeatable. + +To make it harder to do this by accident, redo could intentionally wipe all +but a short whitelist of allowed environment variables before running any +.do script. + +As a bonus, by never sharing any state outside the filesystem, it becomes +much more possible to make a "distributed redo" that builds different +targets on different physical computers. + + +### redo-recheck command + +Normally, redo only checks any given file dependency at most once per +session, in order to reduce the number of system calls executed, thus +greatly speeding up incremental builds. As a result, `redo-ifchange` of the +same target will only execute the relevant .do script at most once per +session. + +In some situations, notably integration tests, we want to force redo to +re-check more often. Right now there's a hacky script called +`t/flush-cache` in the redo distribution which does this, but it relies on +specific knowledge of the .redo directory's database format, which means it +only works in this specific version of redo; this prevents the integration +tests from running (and thus checking compatibility with) competing redo +implementations. + +If we standardized a `redo-recheck` command, which would flush the cache for +the targets given on the command line, and all of their dependencies, this +sort of integration test could work across multiple redo versions. For redo +versions which don't bother caching, `redo-recheck` could be a null +operation. + + +### tty input + +Right now, redo only allows a .do file to request input from the user's +terminal if using `--no-log` and *not* using the `-j` option. Terminal +input is occasionally useful for `make config` interfaces, but parallelism +and log linearization make the console too cluttered for a UI to work. + +The ninja build system has a [console +pool](https://ninja-build.org/manual.html#_the_literal_console_literal_pool) +that can contain up to one job at a time. When a job is in the console +pool, it takes over the console entirely. + +We could probably implement something similar in redo by using POSIX job +control features, which suspend subtasks whenever they try to read +from the tty. If we caught the suspension signal and acquired a lock, we +could serialize console access. + +Whether the complexity of this feature is worthwhile is unclear. Maybe it +makes more sense just to have a './configure' script that runs outside the +redo environment, but still can call into redo-ifchange if needed. + + +### redo-lock command + +Because it supports parallelism via recursion, redo automatically handles +inter-process locking so that only one instance of redo can try to build a +given target at a time. + +This sort of locking turns out to be very useful, but there are a few +situations where requiring redo to "build a target by calling a .do file" in +order to acquire a lock becomes awkward. + +For example, imagine redo is being used to call into `make` to run arbitrary +`Makefile` targets. `default.make.do` might look like this: +```sh +make "$2" +``` + +redo will automatically prevent two copies of `redo all.make` from running +at once. However, if someone runs `redo all.make myprogram.make`, then two +copies of `make` will execute at once. This *might* be harmless, but if the +`all` target in the `Makefile` has a dependency on `myprogram`, then we will +actually end up implicitly building `myprogram` from two places at once: +from the `myprogram` part of `all.make` and from `myprogram.make`. + +In hairy situations like that, it would be nice to serialize all access +inside `default.make.do`, perhaps like this: +```sh +redo-lock make.lock make "$2" +``` + +This would create a redo-style lock on the (virtual) file `make.lock`, but +then instead of trying to `redo make.lock`, it would run the given command, +in this case `make "$2"`. + +It's unclear whether this feature is really a good idea. There are other +(convoluted) ways to achieve the same goal. Nevertheless, it would be easy +enough to implement. And redo versions that don't support parallelism could +just make redo-lock a no-op, since they guarantee serialization in all cases +anyway. + + +### Include a (minimal) POSIX shell + +A common problem when writing build scripts, both in `make` and in redo, is +gratuitous incompatibility between all the available POSIX-like unix shells. +Nowadays, most shells support [various pure POSIX sh +features](https://apenwarr.ca/log/20110228), but there are always glitches. +In some cases, POSIX doesn't define the expected behaviour for certain +situations. In others, shells like `bash` try to "improve" things by +changing the syntax in non-POSIX ways. Or maybe they just add new +backward-compatible features, which you then rely on accidentally because +you only tested your scripts with `bash`. + +redo on Windows using something like [MSYS](http://www.mingw.org/wiki/msys) +is especially limited by the lack of (and oddity of) available unix tools. + +To avoid all these portability problems for .do script maintainers, we might +consider bundling redo with a particular (optional) sh implementation, and +maybe also unix-like tools, that it will use by default. An obvious +candidate would be busybox, which has a win32 version called +[busybox-w32](https://frippery.org/busybox/). + + +### redoconf + +redo is fundamentally a low-level tool that doesn't know as much about +compiling specific programming languages as do higher-level tools like +[cmake](https://cmake.org/). + +Similarly, `make` doesn't know much about specific programming languages +(and what it does know is hopelessly out of date, but cannot be deleted or +updated because it would break backward compatibility with old Makefiles). +This is why `autoconf` and `automake` were created: to automatically fill in +the language- and platform-specific blanks, while letting `make` still +handle executing the low level instructions. + +It might be useful to have a redo-native autoconf/automake-like system, +although you can already use autoconf with redo, so this might not be +essential. diff --git a/mkdocs.yml b/mkdocs.yml index feeef51..f155c3e 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -14,6 +14,7 @@ pages: - Introduction: index.md - Getting Started: GettingStarted.md - Contributing.md + - Roadmap.md - Cookbook: - Hello World (hello.do, redo-ifchange): cookbook/hello/index.md - Text processing example (default.do, redo-whichdo, redo-always, redo-stamp): cookbook/defaults/index.md