forks/apenwarr-redo
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Add README.md and LICENSE.

Browse source 
LGPLv2, by the way.
This commit is contained in:
Avery Pennarun 2010-11-17 00:40:06 -08:00
commit abbde40a4f
3 changed files with 1207 additions and 0 deletions
Download patch file Download diff file Expand all files Collapse all files

481
LICENSE Normal file
View file

@ -0,0 +1,481 @@
GNU LIBRARY GENERAL PUBLIC LICENSE
Version 2, June 1991
Copyright (C) 1991 Free Software Foundation, Inc.
675 Mass Ave, Cambridge, MA 02139, USA
Everyone is permitted to copy and distribute verbatim copies
of this license document, but changing it is not allowed.
[This is the first released version of the library GPL. It is
numbered 2 because it goes with version 2 of the ordinary GPL.]
Preamble
The licenses for most software are designed to take away your
freedom to share and change it. By contrast, the GNU General Public
Licenses are intended to guarantee your freedom to share and change
free software--to make sure the software is free for all its users.
This license, the Library General Public License, applies to some
specially designated Free Software Foundation software, and to any
other libraries whose authors decide to use it. You can use it for
your libraries, too.
When we speak of free software, we are referring to freedom, not
price. Our General Public Licenses are designed to make sure that you
have the freedom to distribute copies of free software (and charge for
this service if you wish), that you receive source code or can get it
if you want it, that you can change the software or use pieces of it
in new free programs; and that you know you can do these things.
To protect your rights, we need to make restrictions that forbid
anyone to deny you these rights or to ask you to surrender the rights.
These restrictions translate to certain responsibilities for you if
you distribute copies of the library, or if you modify it.
For example, if you distribute copies of the library, whether gratis
or for a fee, you must give the recipients all the rights that we gave
you. You must make sure that they, too, receive or can get the source
code. If you link a program with the library, you must provide
complete object files to the recipients so that they can relink them
with the library, after making changes to the library and recompiling
it. And you must show them these terms so they know their rights.
Our method of protecting your rights has two steps: (1) copyright
the library, and (2) offer you this license which gives you legal
permission to copy, distribute and/or modify the library.
Also, for each distributor's protection, we want to make certain
that everyone understands that there is no warranty for this free
library. If the library is modified by someone else and passed on, we
want its recipients to know that what they have is not the original
version, so that any problems introduced by others will not reflect on
the original authors' reputations.
Finally, any free program is threatened constantly by software
patents. We wish to avoid the danger that companies distributing free
software will individually obtain patent licenses, thus in effect
transforming the program into proprietary software. To prevent this,
we have made it clear that any patent must be licensed for everyone's
free use or not licensed at all.
Most GNU software, including some libraries, is covered by the ordinary
GNU General Public License, which was designed for utility programs. This
license, the GNU Library General Public License, applies to certain
designated libraries. This license is quite different from the ordinary
one; be sure to read it in full, and don't assume that anything in it is
the same as in the ordinary license.
The reason we have a separate public license for some libraries is that
they blur the distinction we usually make between modifying or adding to a
program and simply using it. Linking a program with a library, without
changing the library, is in some sense simply using the library, and is
analogous to running a utility program or application program. However, in
a textual and legal sense, the linked executable is a combined work, a
derivative of the original library, and the ordinary General Public License
treats it as such.
Because of this blurred distinction, using the ordinary General
Public License for libraries did not effectively promote software
sharing, because most developers did not use the libraries. We
concluded that weaker conditions might promote sharing better.
However, unrestricted linking of non-free programs would deprive the
users of those programs of all benefit from the free status of the
libraries themselves. This Library General Public License is intended to
permit developers of non-free programs to use free libraries, while
preserving your freedom as a user of such programs to change the free
libraries that are incorporated in them. (We have not seen how to achieve
this as regards changes in header files, but we have achieved it as regards
changes in the actual functions of the Library.) The hope is that this
will lead to faster development of free libraries.
The precise terms and conditions for copying, distribution and
modification follow. Pay close attention to the difference between a
"work based on the library" and a "work that uses the library". The
former contains code derived from the library, while the latter only
works together with the library.
Note that it is possible for a library to be covered by the ordinary
General Public License rather than by this special one.
GNU LIBRARY GENERAL PUBLIC LICENSE
TERMS AND CONDITIONS FOR COPYING, DISTRIBUTION AND MODIFICATION
0. This License Agreement applies to any software library which
contains a notice placed by the copyright holder or other authorized
party saying it may be distributed under the terms of this Library
General Public License (also called "this License"). Each licensee is
addressed as "you".
A "library" means a collection of software functions and/or data
prepared so as to be conveniently linked with application programs
(which use some of those functions and data) to form executables.
The "Library", below, refers to any such software library or work
which has been distributed under these terms. A "work based on the
Library" means either the Library or any derivative work under
copyright law: that is to say, a work containing the Library or a
portion of it, either verbatim or with modifications and/or translated
straightforwardly into another language. (Hereinafter, translation is
included without limitation in the term "modification".)
"Source code" for a work means the preferred form of the work for
making modifications to it. For a library, complete source code means
all the source code for all modules it contains, plus any associated
interface definition files, plus the scripts used to control compilation
and installation of the library.
Activities other than copying, distribution and modification are not
covered by this License; they are outside its scope. The act of
running a program using the Library is not restricted, and output from
such a program is covered only if its contents constitute a work based
on the Library (independent of the use of the Library in a tool for
writing it). Whether that is true depends on what the Library does
and what the program that uses the Library does.
1. You may copy and distribute verbatim copies of the Library's
complete source code as you receive it, in any medium, provided that
you conspicuously and appropriately publish on each copy an
appropriate copyright notice and disclaimer of warranty; keep intact
all the notices that refer to this License and to the absence of any
warranty; and distribute a copy of this License along with the
Library.
You may charge a fee for the physical act of transferring a copy,
and you may at your option offer warranty protection in exchange for a
fee.
2. You may modify your copy or copies of the Library or any portion
of it, thus forming a work based on the Library, and copy and
distribute such modifications or work under the terms of Section 1
above, provided that you also meet all of these conditions:
a) The modified work must itself be a software library.
b) You must cause the files modified to carry prominent notices
stating that you changed the files and the date of any change.
c) You must cause the whole of the work to be licensed at no
charge to all third parties under the terms of this License.
d) If a facility in the modified Library refers to a function or a
table of data to be supplied by an application program that uses
the facility, other than as an argument passed when the facility
is invoked, then you must make a good faith effort to ensure that,
in the event an application does not supply such function or
table, the facility still operates, and performs whatever part of
its purpose remains meaningful.
(For example, a function in a library to compute square roots has
a purpose that is entirely well-defined independent of the
application. Therefore, Subsection 2d requires that any
application-supplied function or table used by this function must
be optional: if the application does not supply it, the square
root function must still compute square roots.)
These requirements apply to the modified work as a whole. If
identifiable sections of that work are not derived from the Library,
and can be reasonably considered independent and separate works in
themselves, then this License, and its terms, do not apply to those
sections when you distribute them as separate works. But when you
distribute the same sections as part of a whole which is a work based
on the Library, the distribution of the whole must be on the terms of
this License, whose permissions for other licensees extend to the
entire whole, and thus to each and every part regardless of who wrote
it.
Thus, it is not the intent of this section to claim rights or contest
your rights to work written entirely by you; rather, the intent is to
exercise the right to control the distribution of derivative or
collective works based on the Library.
In addition, mere aggregation of another work not based on the Library
with the Library (or with a work based on the Library) on a volume of
a storage or distribution medium does not bring the other work under
the scope of this License.
3. You may opt to apply the terms of the ordinary GNU General Public
License instead of this License to a given copy of the Library. To do
this, you must alter all the notices that refer to this License, so
that they refer to the ordinary GNU General Public License, version 2,
instead of to this License. (If a newer version than version 2 of the
ordinary GNU General Public License has appeared, then you can specify
that version instead if you wish.) Do not make any other change in
these notices.
Once this change is made in a given copy, it is irreversible for
that copy, so the ordinary GNU General Public License applies to all
subsequent copies and derivative works made from that copy.
This option is useful when you wish to copy part of the code of
the Library into a program that is not a library.
4. You may copy and distribute the Library (or a portion or
derivative of it, under Section 2) in object code or executable form
under the terms of Sections 1 and 2 above provided that you accompany
it with the complete corresponding machine-readable source code, which
must be distributed under the terms of Sections 1 and 2 above on a
medium customarily used for software interchange.
If distribution of object code is made by offering access to copy
from a designated place, then offering equivalent access to copy the
source code from the same place satisfies the requirement to
distribute the source code, even though third parties are not
compelled to copy the source along with the object code.
5. A program that contains no derivative of any portion of the
Library, but is designed to work with the Library by being compiled or
linked with it, is called a "work that uses the Library". Such a
work, in isolation, is not a derivative work of the Library, and
therefore falls outside the scope of this License.
However, linking a "work that uses the Library" with the Library
creates an executable that is a derivative of the Library (because it
contains portions of the Library), rather than a "work that uses the
library". The executable is therefore covered by this License.
Section 6 states terms for distribution of such executables.
When a "work that uses the Library" uses material from a header file
that is part of the Library, the object code for the work may be a
derivative work of the Library even though the source code is not.
Whether this is true is especially significant if the work can be
linked without the Library, or if the work is itself a library. The
threshold for this to be true is not precisely defined by law.
If such an object file uses only numerical parameters, data
structure layouts and accessors, and small macros and small inline
functions (ten lines or less in length), then the use of the object
file is unrestricted, regardless of whether it is legally a derivative
work. (Executables containing this object code plus portions of the
Library will still fall under Section 6.)
Otherwise, if the work is a derivative of the Library, you may
distribute the object code for the work under the terms of Section 6.
Any executables containing that work also fall under Section 6,
whether or not they are linked directly with the Library itself.
6. As an exception to the Sections above, you may also compile or
link a "work that uses the Library" with the Library to produce a
work containing portions of the Library, and distribute that work
under terms of your choice, provided that the terms permit
modification of the work for the customer's own use and reverse
engineering for debugging such modifications.
You must give prominent notice with each copy of the work that the
Library is used in it and that the Library and its use are covered by
this License. You must supply a copy of this License. If the work
during execution displays copyright notices, you must include the
copyright notice for the Library among them, as well as a reference
directing the user to the copy of this License. Also, you must do one
of these things:
a) Accompany the work with the complete corresponding
machine-readable source code for the Library including whatever
changes were used in the work (which must be distributed under
Sections 1 and 2 above); and, if the work is an executable linked
with the Library, with the complete machine-readable "work that
uses the Library", as object code and/or source code, so that the
user can modify the Library and then relink to produce a modified
executable containing the modified Library. (It is understood
that the user who changes the contents of definitions files in the
Library will not necessarily be able to recompile the application
to use the modified definitions.)
b) Accompany the work with a written offer, valid for at
least three years, to give the same user the materials
specified in Subsection 6a, above, for a charge no more
than the cost of performing this distribution.
c) If distribution of the work is made by offering access to copy
from a designated place, offer equivalent access to copy the above
specified materials from the same place.
d) Verify that the user has already received a copy of these
materials or that you have already sent this user a copy.
For an executable, the required form of the "work that uses the
Library" must include any data and utility programs needed for
reproducing the executable from it. However, as a special exception,
the source code distributed need not include anything that is normally
distributed (in either source or binary form) with the major
components (compiler, kernel, and so on) of the operating system on
which the executable runs, unless that component itself accompanies
the executable.
It may happen that this requirement contradicts the license
restrictions of other proprietary libraries that do not normally
accompany the operating system. Such a contradiction means you cannot
use both them and the Library together in an executable that you
distribute.
7. You may place library facilities that are a work based on the
Library side-by-side in a single library together with other library
facilities not covered by this License, and distribute such a combined
library, provided that the separate distribution of the work based on
the Library and of the other library facilities is otherwise
permitted, and provided that you do these two things:
a) Accompany the combined library with a copy of the same work
based on the Library, uncombined with any other library
facilities. This must be distributed under the terms of the
Sections above.
b) Give prominent notice with the combined library of the fact
that part of it is a work based on the Library, and explaining
where to find the accompanying uncombined form of the same work.
8. You may not copy, modify, sublicense, link with, or distribute
the Library except as expressly provided under this License. Any
attempt otherwise to copy, modify, sublicense, link with, or
distribute the Library is void, and will automatically terminate your
rights under this License. However, parties who have received copies,
or rights, from you under this License will not have their licenses
terminated so long as such parties remain in full compliance.
9. You are not required to accept this License, since you have not
signed it. However, nothing else grants you permission to modify or
distribute the Library or its derivative works. These actions are
prohibited by law if you do not accept this License. Therefore, by
modifying or distributing the Library (or any work based on the
Library), you indicate your acceptance of this License to do so, and
all its terms and conditions for copying, distributing or modifying
the Library or works based on it.
10. Each time you redistribute the Library (or any work based on the
Library), the recipient automatically receives a license from the
original licensor to copy, distribute, link with or modify the Library
subject to these terms and conditions. You may not impose any further
restrictions on the recipients' exercise of the rights granted herein.
You are not responsible for enforcing compliance by third parties to
this License.
11. If, as a consequence of a court judgment or allegation of patent
infringement or for any other reason (not limited to patent issues),
conditions are imposed on you (whether by court order, agreement or
otherwise) that contradict the conditions of this License, they do not
excuse you from the conditions of this License. If you cannot
distribute so as to satisfy simultaneously your obligations under this
License and any other pertinent obligations, then as a consequence you
may not distribute the Library at all. For example, if a patent
license would not permit royalty-free redistribution of the Library by
all those who receive copies directly or indirectly through you, then
the only way you could satisfy both it and this License would be to
refrain entirely from distribution of the Library.
If any portion of this section is held invalid or unenforceable under any
particular circumstance, the balance of the section is intended to apply,
and the section as a whole is intended to apply in other circumstances.
It is not the purpose of this section to induce you to infringe any
patents or other property right claims or to contest validity of any
such claims; this section has the sole purpose of protecting the
integrity of the free software distribution system which is
implemented by public license practices. Many people have made
generous contributions to the wide range of software distributed
through that system in reliance on consistent application of that
system; it is up to the author/donor to decide if he or she is willing
to distribute software through any other system and a licensee cannot
impose that choice.
This section is intended to make thoroughly clear what is believed to
be a consequence of the rest of this License.
12. If the distribution and/or use of the Library is restricted in
certain countries either by patents or by copyrighted interfaces, the
original copyright holder who places the Library under this License may add
an explicit geographical distribution limitation excluding those countries,
so that distribution is permitted only in or among countries not thus
excluded. In such case, this License incorporates the limitation as if
written in the body of this License.
13. The Free Software Foundation may publish revised and/or new
versions of the Library General Public License from time to time.
Such new versions will be similar in spirit to the present version,
but may differ in detail to address new problems or concerns.
Each version is given a distinguishing version number. If the Library
specifies a version number of this License which applies to it and
"any later version", you have the option of following the terms and
conditions either of that version or of any later version published by
the Free Software Foundation. If the Library does not specify a
license version number, you may choose any version ever published by
the Free Software Foundation.
14. If you wish to incorporate parts of the Library into other free
programs whose distribution conditions are incompatible with these,
write to the author to ask for permission. For software which is
copyrighted by the Free Software Foundation, write to the Free
Software Foundation; we sometimes make exceptions for this. Our
decision will be guided by the two goals of preserving the free status
of all derivatives of our free software and of promoting the sharing
and reuse of software generally.
NO WARRANTY
15. BECAUSE THE LIBRARY IS LICENSED FREE OF CHARGE, THERE IS NO
WARRANTY FOR THE LIBRARY, TO THE EXTENT PERMITTED BY APPLICABLE LAW.
EXCEPT WHEN OTHERWISE STATED IN WRITING THE COPYRIGHT HOLDERS AND/OR
OTHER PARTIES PROVIDE THE LIBRARY "AS IS" WITHOUT WARRANTY OF ANY
KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE
IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
PURPOSE. THE ENTIRE RISK AS TO THE QUALITY AND PERFORMANCE OF THE
LIBRARY IS WITH YOU. SHOULD THE LIBRARY PROVE DEFECTIVE, YOU ASSUME
THE COST OF ALL NECESSARY SERVICING, REPAIR OR CORRECTION.
16. IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN
WRITING WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MAY MODIFY
AND/OR REDISTRIBUTE THE LIBRARY AS PERMITTED ABOVE, BE LIABLE TO YOU
FOR DAMAGES, INCLUDING ANY GENERAL, SPECIAL, INCIDENTAL OR
CONSEQUENTIAL DAMAGES ARISING OUT OF THE USE OR INABILITY TO USE THE
LIBRARY (INCLUDING BUT NOT LIMITED TO LOSS OF DATA OR DATA BEING
RENDERED INACCURATE OR LOSSES SUSTAINED BY YOU OR THIRD PARTIES OR A
FAILURE OF THE LIBRARY TO OPERATE WITH ANY OTHER SOFTWARE), EVEN IF
SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH
DAMAGES.
END OF TERMS AND CONDITIONS
Appendix: How to Apply These Terms to Your New Libraries
If you develop a new library, and you want it to be of the greatest
possible use to the public, we recommend making it free software that
everyone can redistribute and change. You can do so by permitting
redistribution under these terms (or, alternatively, under the terms of the
ordinary General Public License).
To apply these terms, attach the following notices to the library. It is
safest to attach them to the start of each source file to most effectively
convey the exclusion of warranty; and each file should have at least the
"copyright" line and a pointer to where the full notice is found.
<one line to give the library's name and a brief idea of what it does.>
Copyright (C) <year> <name of author>
This library is free software; you can redistribute it and/or
modify it under the terms of the GNU Library General Public
License as published by the Free Software Foundation; either
version 2 of the License, or (at your option) any later version.
This library is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
Library General Public License for more details.
You should have received a copy of the GNU Library General Public
License along with this library; if not, write to the Free
Software Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.
Also add information on how to contact you by electronic and paper mail.
You should also get your employer (if you work as a programmer) or your
school, if any, to sign a "copyright disclaimer" for the library, if
necessary. Here is a sample; alter the names:
Yoyodyne, Inc., hereby disclaims all copyright interest in the
library `Frob' (a library for tweaking knobs) written by James Random Hacker.
<signature of Ty Coon>, 1 April 1990
Ty Coon, President of Vice
That's all there is to it!

726
README.md Normal file
View file

@ -0,0 +1,726 @@
# redo: a top-down software build system
`redo` is a competitor to the long-lived, but sadly imperfect, `make`
program. There are many such competitors, because many people over the
years have been dissatisfied with make's limitations. However, of all the
replacements I've seen, only redo captures the essential simplicity and
flexibility of make, while eliminating its flaws. To my great surprise, it
manages to do this while being simultaneously simpler than make, more
flexible than make, and more powerful than make.
Although I wrote redo and I would love to take credit for it, this magical
simplicity and flexibility comes because I copied verbatim a design by
Daniel J. Bernstein (creator of qmail and djbdns, among many other useful
things). He posted some very terse notes on his web site at one point
(there is no date) with the unassuming title, "[Rebuilding target files when
source files have changed](http://cr.yp.to/redo.html)." Those notes are
enough information to understand how the system it supposed to work;
unfortunately there's no code to go with it. I get the impression that the
hypothetical "djb redo" is incomplete and Bernstein doesn't yet consider it
ready for the real world.
I was led to that particular page by random chance from a link on [The djb
way](http://thedjbway.b0llix.net/future.html), by Wayne Marshall.
After I found out about djb redo, I searched the Internet for any sign that
other people had discovered what I had: a hidden, unimplemented gem of
brilliant code design. I found only one interesting link: Alan Grosskurth, whose
[Master's thesis at the University of
Waterloo](http://grosskurth.ca/papers/mmath-thesis.pdf) was about top-down
software rebuilding, that is, djb redo. He wrote his own (admittedly slow)
implementation in about 250 lines of shell script.
If you've ever thought about rewriting GNU make from scratch, the idea of
doing it in 250 lines of shell script probably didn't occur to you. redo is
so simple that it's actually possible. For testing, I actually wrote an
even more minimal version (which always rebuilds everything instead of
checking dependencies) in only 70 lines of shell.
The design is simply that good.
This implementation of redo is called `redo` for the same reason that there
are 75 different versions of `make` that are all called `make`. It's just
easier that way. Hopefully it will turn out to be compatible with the other
implementations, should there be any.
# License
My version of redo was written without ever seeing redo code by Bernstein or
Grosskurth, so I own the entire copyright. It's distributed under the GNU
LGPL version 2. You can find a copy of it in the file called LICENSE.
# What's so special about redo?
The theory behind redo is almost magical: it can do everything `make` can
do, only the implementation is vastly simpler, the syntax is cleaner, and you
can do even more flexible things without resorting to ugly hacks. Also, you
get all the speed of non-recursive `make` (only check dependencies once per
run) combined with all the cleanliness of recursive `make` (you don't have
code from one module stomping on code from another module).
But the easiest way to show it is with an example.
Create a file called default.o.do:
redo-ifchange $1.c
gcc -MD -MF deps.tmp -c -o $3 $1.c
DEPS=$(sed -e "s/^$3://" -e 's/\\//g' <deps.tmp)
rm -f deps.tmp
redo-ifchange $DEPS
Create a file called myprog.do:
DEPS="a.o b.o"
redo-ifchange $DEPS
gcc -o $3 $DEPS
Of course, you'll also have to create `a.c` and `b.c`, the C language
source files that you want to build to create your application.
In a.c:
#include <stdio.h>
#include "b.h"
int main() { printf(bstr); }
In b.h:
extern char *bstr;
In b.c:
char *bstr = "hello, world!\n";
Now you simply run:
$ redo myprog
And it says:
redo myprog
redo a.o
redo b.o
Now try this:
$ touch b.h
$ redo myprog
Sure enough, it says:
redo myprog
redo a.o
Did you catch the shell incantation in `default.o.do` where it generates
the autodependencies? The filename `default.o.do` means "run this script to
generate a .o file unless there's a more specific whatever.o.do script that
applies."
The amazing innovation in redo - and really, this is the key innovation that
makes everything else work - is that declaring a dependency is just another
shell command. The `redo-ifchange` command means, "build each of my
arguments. If any of them or their dependencies ever change, then I need to
run the *current script* over again."
Dependencies are tracked in a persistent `.redo` database so that redo can
check them later. If a file needs to be rebuilt, it re-executes the
`whatever.do` script and regenerates the dependencies. If a file doesn't
need to be rebuilt, redo can calculate that just using its persistent
`.redo` database, without re-running the script. And it can do that check
just once right at the start of your project build.
But best of all, as you can see in `default.o.do`, you can declare a
dependency *after* building the program. In C, you get your best dependency
information by trying to actually build, since that's how you find out which
headers you need. redo offers the following simple insight: you don't actually
care what the dependencies are *before* you build the file; if the file
doesn't exist, you obviously need to build it. Then, the build script
itself can provide the dependency information however it wants; unlike in
`make`, you don't need a special dependency syntax at all. You can even
declare your dependencies after building, which makes everything much
simpler.
(GNU make supports putting some of your dependencies in include files, and
auto-reloading those include files if they change. But this is very
confusing - the program flow through a Makefile is hard to trace already,
and even harder if it restarts randomly from the beginning when a file
changes. With redo, you can just read the script from top to bottom. A
`redo-ifchange` call is like calling a function, which you can also read
from top to bottom.)
# One script per file? Can't I just put it all in one big Redofile like make does?
One of my favourite features of redo is that it doesn't add any new syntax;
the syntax of redo is *just* the syntax of sh.
Also, it's surprisingly useful to have each build script in its own file;
that way, you can declare a dependency on just that one build script instead
of the entire Makefile, and you won't have to rebuild everything just
because of a one-line Makefile change. (Some build tools avoid that
by tracking exactly which variables and commands were used to do the build.
But that's more complex, more error prone, and slower.)
Still, it would be rather easy to make a "Redofile" parser that just has a
bunch of sections like this:
myprog:
DEPS="a.o b.o"
redo-ifchange $DEPS
gcc -o $3 $DEPS
We could just auto-extract myprog.do by slurping out the indented sections
into their own files. You could even write a .do file to do it.
It's not obvious that this would be a real improvement however.
See djb's [Target files depend on build
scripts](http://cr.yp.to/redo/honest-script.html) article for more
information.
# Can a *.do file itself be generated as part of the build process?
Not currently. There's nothing fundamentally preventing us from allowing
it. However, it seems easier to reason about your build process if you
*aren't* auto-generating your build scripts on the fly.
This might change.
# Do end users have to have redo installed in order to build my project?
No. We include a very short (70 lines, as of this writing) shell script
called `do` in the `minimal` subdirectory of the redo project. `do` is like
`redo` (and it works with the same `*.do` scripts), except it doesn't
understand dependencies; it just always rebuilds everything from the top.
You can include `do` with your program to make it so non-users of redo can
still build your program. Someone who wants to hack on your program will
probably go crazy unless they have a copy of `redo` though.
Actually, `redo` itself isn't so big, so for large projects where it
matters, you could just include it with your project.
# How does redo store dependencies?
FIXME:
Currently, in a directory called `.redo` that's full of text files. This
isn't really optimal, so it will change eventually. Please consider the
storage format undocumented (but feel free to poke around and look; it's
simple enough).
# If a target didn't change, how to I prevent dependents from being rebuilt?
For example, running ./configure creates a bunch of files including
config.h, and config.h might or might not change from one run to the next.
We don't want to rebuild everything that depends on config.h if config.h is
identical.
With `make`, which makes build decisions based on timestamps, you would
simply have the ./configure script write to config.h.new, then only
overwrite config.h with that if the two files are different.
FIXME:
This is not possible in the current version of `redo`. redo knows whenever it
rebuilds a target and doesn't bother re-checking dependencies after that;
even if the file didn't technically change, it is considered "rebuilt,"
which means all its dependants now need to be rebuilt.
The advantage of this method is you can't accidentally prevent the
rebuilding of things just by marking the target files as "newer" or marking
the source files as "older" (as sometimes happens when you extract an old
tarball or backup on top of your source code files). The disadvantage is
unnecessary rebuilding of some stuff sometimes.
We will have to find a solution to this before redo 1.0.
See also the next question.
# What about checksum-based dependencies instead of timestamp-based ones?
Some build systems keep a checksum of target files and rebuild dependants
only when the target changes. This is appealing in some cases; for example,
with ./configure generating config.h, it could just go ahead and generate
config.h; the build system would be smart enough to rebuild or not rebuild
dependencies automatically. This keeps build scripts simple and gets rid of
the need for people to re-implement file comparison over and over in every
project or for multiple files in the same project.
I think this would be a good addition to `redo` - and not a very difficult
one.
Probably we should add a new command similar to `redo-ifchange`; let's call
it `redo-ifsum` or `redo-ifdiff`. That command would verify checksums
instead of timestamps.
Sometimes you don't want to use checksums for verification; for example, in
some complicated build systems, you want to create empty `something.stamp`
files to indicate that some big complex operation has completed
successfully. But empty files all have the same checksum, so perhaps you'd
rather just use a timestamp comparison in that case. (Alternatively, you
could fill the file with data - maybe a series of checksums - indicating the
state of the data that was produced. If that data changed, the stamp would
then be out of date.)
FIXME: This requires a bit more thought before we commit to any particular
option.
# Can my .do files be written in a language other than sh?
FIXME: Not presently. In theory, we could support starting your .do files
with the magic "#!/" sequence (eg. #!/usr/bin/python) and then using that
shell to run your .do script. But that opens new problems, like figuring
out what is the equivalent of the `redo-ifchange` command in python. Do you
just run it in a subprocess? That might be unnecessarily slow. And so on.
Right now, `redo` explicitly runs `sh -c filename.do`. The main reasons for
this are to make the #!/ line optional, and so you don't have to remember to
`chmod +x` your .do files.
# Can a single .do script generate multiple outputs?
FIXME: Not presently. This seems like a useful feature though.
For example, you could have a file called `default.do.do` that would
generate .do files from a `Redofile`. Then you wouldn't have to argue with
the `redo` maintainers about whether putting stuff into a single `Redofile`
is better than the current behaviour.
Right now you could do that, except you would want to parse the `Redofile`
only once and produce a bunch of `.do` files from that single action. But
you would still want `redo` to know which `.do` files were produced, so it
could rerun the splitter script, if `Redofile` ever changed, before using
one of the generated `.do` files.
It would also be useful, again, with ./configure: typically running the
configure script produces several output files, and it would be nice to
declare dependencies on all of them.
# Recursive make is considered harmful. Isn't redo even *more* recursive?
You probably mean [this 1997 paper](http://miller.emu.id.au/pmiller/books/rmch/)
by Peter Miller.
Yes, redo is recursive, in the sense that every target is built by its own
`.do` file, and every `.do` file is a shell script being run recursively
from other shell scripts, which might call back into `redo`. In fact, it's
even more recursive than recursive make.
However the reason recursive make is considered harmful is that each
instance of make has no access to the dependency information seen by the
other instances. Each one starts from its own Makefile, which only has a
partial picture of what's going on; moreover, each one has to stat a lot of
the same files over again, leading to slowness. That's the thesis of the
"considered harmful" paper.
On the other hand, nobody has written a paper about it, but non-recursive
make should also be considered harmful. The problem is Makefiles aren't
very "hygienic" or "modular"; if you're not running make recursively, then
your one copy of make has to know *everything* about *everything* in your
entire project. Every variable in make is global, so every variable defined
in *any* of your Makefiles is visible in *all* of your Makefiles. Every
little private function or macro is visible everywhere. In a huge project
made up of multiple projects from multiple vendors, that's just not okay.
`redo` deftly manages to dodge both sets of problems. First of all,
dependency information is shared through a global persistent `.redo`
directory, which is accessed by all your `redo` instances at once.
Dependencies created or checked by one instance can be immediately used by
another instance. And there's locking to prevent two instances from
building the same target at the same time. So you get all the "global
dependency" knowledge of non-recursive make.
But also, every `.do` script is entirely hygienic and traceable; `redo`
discourages the use of global environment variables, suggesting that you put
settings into files (which can have timestamps and dependencies) instead.
So you also get all the hygiene and modularity advantages of recursive make.
By the way, you can trace any `redo` build process just by reading the `.do`
scripts from top to bottom. Makefiles are actually a collection of "rules"
whose order of execution is unclear; any rule might run at any time. In a
non-recursive Makefile setup with a bunch of included files, you end up with
lots and lots of rules that can all be executed in a random order; tracing
becomes impossible. Recursive make tries to compensate for this by breaking
the rules into subsections, but that ends up with all the "considered harmful"
paper's complaints. `redo` just runs from top to bottom in a nice tree, so
it's traceable no matter how many layers you have.
# How do I set environment variables that affect the entire build?
Directly using environment variables is a bad idea because you can't declare
dependencies on them. Also, if there were a file that contained a set of
variables that all your .do scripts need to run, then `redo` would have to
read that file every time it starts (which is frequently, since it's
recursive), and that could get slow.
Luckily, there's an alternative. Once you get used to it, this method is
actually much better than environment variables, because it runs faster
*and* it's easier to debug.
For example, djb often uses a computer-generated script called `compile` for
compiling a .c file into a .o file. To generate the `compile` script, we
create a file called `compile.do`:
redo-ifchange config.sh
. ./config.sh
echo "gcc -c -o \$3 $1.c $CFLAGS"
chmod a+x $3
(Note that if a .do script produces data to stdout, like we have here, then
`redo` will use that to atomically update the target file. So you don't
have to redirect it yourself, or worry about using a temp file and then
renaming it.)
Then, your `default.o.do` can simply look like this:
redo-ifchange compile $1.c
./compile $1 $2 $3
This is not only elegant, it's useful too. With make, you have to always
output everything it does to stdout/stderr so you can try to figure out
exactly what it was running; because this gets noisy, some people write
Makefiles that deliberately hide the output and print something friendlier,
like "Compiling hello.c". But then you have to guess what the compile
command looked like.
With redo, the command *is* `./compile hello.c`, which looks good when
printed, but is also completely meaningful. Because it doesn't depend on
any environment variables, you can just run `./compile hello.c` to reproduce
its output, or you can look inside the `compile` file to see exactly what
command line is being used.
As a bonus, all the variable expansions only need to be done once: when
generating the ./compile program. With make, it would be recalculating
expansions every time it compiles a file.
# How do I write a default.o.do that works for both C and C++ source?
See the previous answer for why you will probably use a compile.do instead.
Then your default.o.do looks like it does in the previous answer. We can
then upgrade compile.do to look something like this:
redo-ifchange config.sh
. ./config.sh
cat <<-EOF
[ -e "\$1.cc" ] && EXT=.cc || EXT=.c
gcc -o "\$3" -c "\$1\$EXT" -Wall $CFLAGS
EOF
chmod a+x "$3"
Isn't it expensive to have ./compile doing this kind of test for every
single source file? Not really. Remember, if you have two implicit rules
in make:
%.o: %.cc
gcc ...
%.o: %.c
gcc ...
Then it has to do all the same checks. Except make has even *more* implicit
rules than that, so it ends up trying and discarding lots of possibilities
before it actually builds your program.
So in this case, it's not implicit at all; you're specifying exactly how to
decide whether it's a C program or a C++ program, and what to do in each
case. You can also share the two gcc command lines between the two rules,
which is hard in make. (In GNU make you can use macro functions, but the
syntax for those is ugly.)
# Can I just rebuild a part of the project?
Absolutely! Although `redo` runs "top down" in the sense of one .do file
calling into all its dependencies, you can start at any point in the
dependency tree that you want.
Unlike recursive make, no matter which subdir of your project you're in when
you start, `redo` will be able to build all the dependencies in the right
order.
Unlike non-recursive make, you don't have to jump through any strange hoops
(like adding a fake Makefile in each directory that does `make -C ${TOPDIR}`
back up to the main non-recursive Makefile). redo just uses `filename.do`
to build `filename`, or uses `default*.do` if the specific `filename.do`
doesn't exist.
When running any .do file, `redo` makes sure its current directory is set to
the directory where the .do file is located. That means you can do this:
redo ../utils/foo.o
And it will work exactly like this:
cd ../utils
redo foo.o
# Can my filenames have spaces in them?
Yes, unlike with make. For historical reasons, the Makefile syntax doesn't
support filenames with spaces; spaces are used to separate one filename from
the next, and there's no way to escape these spaces.
Since redo just uses sh, which has working escape characters and
quoting, it doesn't have this problem.
# Does redo care about the differences between tabs and spaces?
No.
# What if my .c file depends on a generated .h file?
This problem arises as follows. foo.c includes config.h, and config.h is
created by running ./configure. The second part is easy; just write a config.h.do
that depends on the existence of configure (which is created by
configure.do, which probably runs autoconf).
The first part, however, is not so easy. Normally, the headers that a C
file depends on are detected as part of the compilation process. That works
fine if the headers, themselves, don't need to be generated first. But if
you do
redo foo.o
There's no way for redo to know that compiling foo.c into foo.o depends on
first generating config.h.
FIXME:
There seem to be a few workarounds for this, but none are very good. You
can explicitly make all .o files depend on config.h (which will cause things
to get recompiled unnecessarily). You can just tell people to run
./configure before running redo (which is what make users typically do).
You can try to guess dependencies using `gcc -MM -MG` (-MG means don't die
if headers aren't present), but that's not actually very helpful, because of
the compiler's header include paths: if I #include "foo.h", which directory
is redo supposed to look at to find foo.h? All of them? Normally a C
file's autodependencies have full paths attached, so this would be special.
[djb has a solution for his own
projects](http://cr.yp.to/redo/honest-nonfile.html), which is a simple
script that looks through C files to pull out #include lines. He assumes
that `#include <file.h>` is a system header (thus not subject to being
built) and `#include "file.h"` is in the current directory (thus easy to
find). Unfortunately this isn't really a complete solution in the general
case.
Maybe just telling redo to try to generate the file in all possible
locations is the best way to go. But we need to think about this more.
In the meantime, you'll have to explicitly specify such dependencies in your
*.o.do files.
# Why don't you by default print the commands as they are run?
make prints the commands it runs as it runs them. redo doesn't, although
you can get this behaviour with `redo -v`.
The main reason we made this decision is that the commands get pretty long
winded (a compiler command line might be multiple lines of repeated
gibberish) and, on large projects, it's hard to actually see the progress of
the overall build. Thus, make users often work hard to have make hide the
command output in order to make the log "more useful."
The reduced output is a pain with make, however, because if there's ever a
problem, you're left wondering exactly what commands were run at what time,
and you often have to go editing the Makefile in order to figure it out.
With redo, it's much less of a problem. By default, redo produces output
that looks like this:
$ redo t
redo t/all
redo t/hello
redo t/LD
redo t/hello.o
redo t/CC
redo t/yellow
redo t/yellow.o
redo t/bellow
redo t/c
redo t/c.c
redo t/c.c.c
redo t/c.c.c.b
redo t/c.c.c.b.b
redo t/d
The indentation indicates the level of recursion (deeper levels are
dependencies of earlier levels). The repeated word "redo" down the left
column looks strange, but it's there for a reason, and the reason is this:
you can cut-and-paste a line from the build script and rerun it directly.
$ redo t/c
redo t/c
redo t/c.c
redo t/c.c.c
redo t/c.c.c.b
redo t/c.c.c.b.b
So if you ever want to debug what happened at a particular step, you can
choose to run only that step in verbose mode:
$ ./redo t/c.c.c.b.b -v
redo t/c.c.c.b.b
redo-ifchange "$1$2.a"
echo a-to-b
cat "$1$2.a"
If you're using an autobuilder or something that logs build results for
future examination, you should probably set it to always run redo with
the -v option.
# Is redo compatible with autoconf?
Yes. You don't have to do anything special, other than the above note about
declaring dependencies on config.h, which is no worse than what you would
have to do with make.
# Is redo compatible with automake?
Hells no. You can thank me later. But see next question.
# Is redo compatible with make?
Yes. If you have an existing Makefile (for example, in one of your
subprojects), you can just call make to build that subproject.
In a file called myproject.stamp.do:
redo-ifchange $(find myproject -name '*.[ch]')
make -C myproject all
So, to amend our answer to the previous question, you can include
automake-generated Makefiles as part of your redo-based project.
# Is redo -j compatible with make -j?
Yes! redo implements the same jobserver protocol as GNU make, which means
that redo running under make -j, or make running under redo -j, will do the
right thing. Thus, it's safe to mix-and-match redo and make in a recursive
build system. Just make sure you declare your dependencies correctly.
# What about distributed builds?
FIXME:
So far, nobody has tried redo in a distributed build environment. It surely
works with distcc, since that's just a distributed compiler. But there are
other systems that distribute more of the build process to other machines.
The most interesting method I've heard of was explained (in public, this is
not proprietary information) by someone from Google. Apparently, the
Android team uses a tool that mounts your entire local filesystem on a
remote machine using FUSE and chroots into that directory. Then you replace
the $SHELL variable in your copy of make with one that runs this tool.
Because the remote filesystem is identical to yours, the build will
certainly complete successfully. After the $SHELL program exits, the changed
files are sent back to your local machine. Cleverly, the files on the
remote server are cached based on their checksums, so files only need to be
re-sent if they have changed since last time. This dramatically reduces
bandwidth usage compared to, say, distcc (which mostly just re-sends the
same preparsed headers over and over again).
At the time, he promised to open source this tool eventually. It would be
pretty fun to play with it.
FIXME: However, it won't work as easily with redo as it did with make. With
make, a separate copy of $SHELL is launched for each step of the build (and
gets migrated to the remote machine), but make runs only on your local
machine, so it can control parallelism and avoid building the same target
from multiple machines, and so on. With redo, since the entire script runs
inside the shell, we'd have to do the parallelism another way.
# How fast is redo compared to make?
FIXME:
The current version of redo is written in python and has not been optimized.
So right now, it's usually a bit slower. Not too embarrassingly slower,
though, and the slowness really only strikes the first time you build a
project.
For incrementally building only the changed parts of the project, redo can
be much faster than make, because it can check all the dependencies up
front and doesn't need to repeatedly parse and re-parse the Makefile (as
recursive make needs to do).
More bad news, however: the current version of redo also pointlessly
re-evaluates the same dependencies over and over. Unlike with make, there
is no good reason for this, so it'll be fixed eventually.
Even better, in 'redo -j' mode, it sometimes rebuilds the
same dependency more than once. Not at the same time, so
your build won't break, but it'll just be unnecessarily
slow. Obviously this should be fixed.
Furthermore, redo's current dependency database (in the `.redo` directory)
is not very efficient, and thrashes a lot of filesystem metadata, which is
especially slow on some filesystems, notably ext3. We'll want to improve that
by using a "real" data structure eventually. With a "real" data structure,
a daemon, and inotify, you could actually get redo's dependency evaluation
to run instantly. On very large projects with tens of thousands of files to
stat() before we can even start building, that could make a pretty big
difference. With make, an inotify implementation would be pretty hard to
do (since just parsing the Makefiles of such a project is complicated, and
there's no guarantee the dependencies are the same as last time). But with
the way redo works, this kind of optimization would be pretty easy.
We'll probably have to rewrite redo in C eventually to speed it up further.
This won't be very hard; the design is so simple that it should be easy to
write in any language. It's just *even easier* in python,
which was good for writing a prototype.
Most of the so-called slowness at the moment (it's really not that bad)
is because redo-ifchange (and also sh itself) need to be fork'd and
exec'd over and over during the build process. The `minimal/do` script
shows a way around this; redo-ifchange could be a shell function
instead of a standalone program. Eliminating the need for extra
fork/exec in the common case could actually get us much faster than
make even when doing an initial build; make executes $SHELL for every
command, but with redo, there is a shell running at all times anyway,
so if we're very careful we could optimize that out.
As a point of reference, on my computer, I can fork-exec
redo-ifchange.py about 87 times per second; an empty python program,
about 100 times per second; an empty C program, about 1000 times per
second; an empty make, about 300 times per second. So if I could
compile 87 files per second, which I can't, then python overhead
would be 50%. Also, if you're using redo -j on a multicore machine, all
the python forking happens in parallel with everything else, so that's
87 per second per core. Nevertheless, that's still slower than make and
should be fixed.
# What's missing? How can I help?
redo is thoroughly incomplete and probably has numerous bugs. Just what you
always wanted in a build system, I know.
What's missing? Search for the word FIXME in this document; anything with a
FIXME is something that is either not implemented, or which needs
discussion, feedback, or ideas. Of course, there are surely other
undocumented things that need discussion or fixes too.
You should join the redo-list@googlegroups.com mailing list.
You can find the mailing list archives here:
http://groups.google.com/group/redo-list
Yes, it might not look like it, but you can subscribe without having a
Google Account. Just send a message here:
redo-list+subscribe@googlegroups.com
Note the plus sign.
Have fun,
Avery

0
t/hello.c Executable file → Normal file
View file