forks/apenwarr-redo
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
apenwarr-redo/redo/vars_init.py

53 lines
1.7 KiB
Python
Raw Normal View History

redo-ifchange can now be run even if there's no parent redo.
2010-12-11 19:08:53 -08:00
import sys, os
redo-whichdo: fix a bug where the last dir was checked twice, and add tests. When we can't find a .do file, we walk all the way back to the root directory. When that happens, the root directory is actually searched twice. This is harmless (since a .do file doesn't exist there anyway) but causes redo-whichdo to produce the wrong output. Also, add a test, which I forgot to do when writing whichdo in the first place. To make the test work from the root directory, we need a way to initialize redo without actually creating a .redo directory. Add a init_no_state() function for that purpose, and split the necessary path functions into their own module so we can avoid importing builder.py.
2018-10-30 23:23:04 -04:00
redo-log: capture and linearize the output of redo builds. redo now saves the stderr from every .do script, for every target, into a file in the .redo directory. That means you can look up the logs from the most recent build of any target using the new redo-log command, for example: redo-log -r all The default is to show logs non-recursively, that is, it'll show when a target does redo-ifchange on another target, but it won't recurse into the logs for the latter target. With -r (recursive), it does. With -u (unchanged), it does even if redo-ifchange discovered that the target was already up-to-date; in that case, it prints the logs of the *most recent* time the target was generated. With --no-details, redo-log will show only the 'redo' lines, not the other log messages. For very noisy build systems (like recursing into a 'make' instance) this can be helpful to get an overview of what happened, without all the cruft. You can use the -f (follow) option like tail -f, to follow a build that's currently in progress until it finishes. redo itself spins up a copy of redo-log -r -f while it runs, so you can see what's going on. Still broken in this version: - No man page or new tests yet. - ANSI colors don't yet work (unless you use --raw-logs, which gives the old-style behaviour). - You can't redirect the output of a sub-redo to a file or a pipe right now, because redo-log is eating it. - The regex for matching 'redo' lines in the log is very gross. Instead, we should put the raw log files in a more machine-parseable format, and redo-log should turn that into human-readable format. - redo-log tries to "linearize" the logs, which makes them comprehensible even for a large parallel build. It recursively shows log messages for each target in depth-first tree order (by tracing into a new target every time it sees a 'redo' line). This works really well, but in some specific cases, the "topmost" redo instance can get stuck waiting for a jwack token, which makes it look like the whole build has stalled, when really redo-log is just waiting a long time for a particular subprocess to be able to continue. We'll need to add a specific workaround for that.
2018-11-03 22:09:18 -04:00
is_toplevel = False
redo-whichdo: fix a bug where the last dir was checked twice, and add tests. When we can't find a .do file, we walk all the way back to the root directory. When that happens, the root directory is actually searched twice. This is harmless (since a .do file doesn't exist there anyway) but causes redo-whichdo to produce the wrong output. Also, add a test, which I forgot to do when writing whichdo in the first place. To make the test work from the root directory, we need a way to initialize redo without actually creating a .redo directory. Add a init_no_state() function for that purpose, and split the necessary path functions into their own module so we can avoid importing builder.py.
2018-10-30 23:23:04 -04:00
def init_no_state():
Move into the 21st century by fixing some pylint warnings.
2018-12-02 23:15:37 -05:00
global is_toplevel
redo-whichdo: fix a bug where the last dir was checked twice, and add tests. When we can't find a .do file, we walk all the way back to the root directory. When that happens, the root directory is actually searched twice. This is harmless (since a .do file doesn't exist there anyway) but causes redo-whichdo to produce the wrong output. Also, add a test, which I forgot to do when writing whichdo in the first place. To make the test work from the root directory, we need a way to initialize redo without actually creating a .redo directory. Add a init_no_state() function for that purpose, and split the necessary path functions into their own module so we can avoid importing builder.py.
2018-10-30 23:23:04 -04:00
if not os.environ.get('REDO'):
os.environ['REDO'] = 'NOT_DEFINED'
redo-log: capture and linearize the output of redo builds. redo now saves the stderr from every .do script, for every target, into a file in the .redo directory. That means you can look up the logs from the most recent build of any target using the new redo-log command, for example: redo-log -r all The default is to show logs non-recursively, that is, it'll show when a target does redo-ifchange on another target, but it won't recurse into the logs for the latter target. With -r (recursive), it does. With -u (unchanged), it does even if redo-ifchange discovered that the target was already up-to-date; in that case, it prints the logs of the *most recent* time the target was generated. With --no-details, redo-log will show only the 'redo' lines, not the other log messages. For very noisy build systems (like recursing into a 'make' instance) this can be helpful to get an overview of what happened, without all the cruft. You can use the -f (follow) option like tail -f, to follow a build that's currently in progress until it finishes. redo itself spins up a copy of redo-log -r -f while it runs, so you can see what's going on. Still broken in this version: - No man page or new tests yet. - ANSI colors don't yet work (unless you use --raw-logs, which gives the old-style behaviour). - You can't redirect the output of a sub-redo to a file or a pipe right now, because redo-log is eating it. - The regex for matching 'redo' lines in the log is very gross. Instead, we should put the raw log files in a more machine-parseable format, and redo-log should turn that into human-readable format. - redo-log tries to "linearize" the logs, which makes them comprehensible even for a large parallel build. It recursively shows log messages for each target in depth-first tree order (by tracing into a new target every time it sees a 'redo' line). This works really well, but in some specific cases, the "topmost" redo instance can get stuck waiting for a jwack token, which makes it look like the whole build has stalled, when really redo-log is just waiting a long time for a particular subprocess to be able to continue. We'll need to add a specific workaround for that.
2018-11-03 22:09:18 -04:00
is_toplevel = True
redo-whichdo: fix a bug where the last dir was checked twice, and add tests. When we can't find a .do file, we walk all the way back to the root directory. When that happens, the root directory is actually searched twice. This is harmless (since a .do file doesn't exist there anyway) but causes redo-whichdo to produce the wrong output. Also, add a test, which I forgot to do when writing whichdo in the first place. To make the test work from the root directory, we need a way to initialize redo without actually creating a .redo directory. Add a init_no_state() function for that purpose, and split the necessary path functions into their own module so we can avoid importing builder.py.
2018-10-30 23:23:04 -04:00
if not os.environ.get('REDO_BASE'):
os.environ['REDO_BASE'] = 'NOT_DEFINED'
redo-ifchange can now be run even if there's no parent redo.
2010-12-11 19:08:53 -08:00
def init(targets):
Move into the 21st century by fixing some pylint warnings.
2018-12-02 23:15:37 -05:00
global is_toplevel
redo-ifchange can now be run even if there's no parent redo.
2010-12-11 19:08:53 -08:00
if not os.environ.get('REDO'):
# toplevel call to redo
redo-log: capture and linearize the output of redo builds. redo now saves the stderr from every .do script, for every target, into a file in the .redo directory. That means you can look up the logs from the most recent build of any target using the new redo-log command, for example: redo-log -r all The default is to show logs non-recursively, that is, it'll show when a target does redo-ifchange on another target, but it won't recurse into the logs for the latter target. With -r (recursive), it does. With -u (unchanged), it does even if redo-ifchange discovered that the target was already up-to-date; in that case, it prints the logs of the *most recent* time the target was generated. With --no-details, redo-log will show only the 'redo' lines, not the other log messages. For very noisy build systems (like recursing into a 'make' instance) this can be helpful to get an overview of what happened, without all the cruft. You can use the -f (follow) option like tail -f, to follow a build that's currently in progress until it finishes. redo itself spins up a copy of redo-log -r -f while it runs, so you can see what's going on. Still broken in this version: - No man page or new tests yet. - ANSI colors don't yet work (unless you use --raw-logs, which gives the old-style behaviour). - You can't redirect the output of a sub-redo to a file or a pipe right now, because redo-log is eating it. - The regex for matching 'redo' lines in the log is very gross. Instead, we should put the raw log files in a more machine-parseable format, and redo-log should turn that into human-readable format. - redo-log tries to "linearize" the logs, which makes them comprehensible even for a large parallel build. It recursively shows log messages for each target in depth-first tree order (by tracing into a new target every time it sees a 'redo' line). This works really well, but in some specific cases, the "topmost" redo instance can get stuck waiting for a jwack token, which makes it look like the whole build has stalled, when really redo-log is just waiting a long time for a particular subprocess to be able to continue. We'll need to add a specific workaround for that.
2018-11-03 22:09:18 -04:00
is_toplevel = True
redo: only default to 'all' in the toplevel instance of redo. We already did this in minimal/do, and redo-ifchange already did this, but plain redo didn't. This made constructs like: for d in *.x; do echo "${d%.x}" done | xargs redo dangerous, because if there were no matching files, we'd try to 'redo all'.
2011-03-10 21:10:15 -08:00
if len(targets) == 0:
targets.append('all')
redo-ifchange can now be run even if there's no parent redo.
2010-12-11 19:08:53 -08:00
exenames = [os.path.abspath(sys.argv[0]),
os.path.realpath(sys.argv[0])]
dirnames = [os.path.dirname(p) for p in exenames]
Automatically select a good shell instead of relying on /bin/sh. This includes a fairly detailed test of various known shell bugs from the autoconf docs. The idea here is that if redo works on your system, you should be able to rely on a *good* shell to run your .do files; you shouldn't have to work around zillions of bugs like autoconf does.
2010-12-21 04:19:50 -08:00
trynames = ([os.path.abspath(p+'/../lib/redo') for p in dirnames] +
Directory reorg: move code into redo/, generate binaries in bin/. It's time to start preparing for a version of redo that doesn't work unless we build it first (because it will rely on C modules, and eventually be rewritten in C altogether). To get rolling, remove the old-style symlinks to the main programs, and rename those programs from redo-*.py to redo/cmd_*.py. We'll also move all library functions into the redo/ dir, which is a more python-style naming convention. Previously, install.do was generating wrappers for installing in /usr/bin, which extend sys.path and then import+run the right file. This made "installed" redo work quite differently from running redo inside its source tree. Instead, let's always generate the wrappers in bin/, and not make anything executable except those wrappers. Since we're generating wrappers anyway, let's actually auto-detect the right version of python for the running system; distros can't seem to agree on what to call their python2 binaries (sigh). We'll fill in the right #! shebang lines. Since we're doing that, we can stop using /usr/bin/env, which will a) make things slightly faster, and b) let us use "python -S", which tells python not to load a bunch of extra crap we're not using, thus improving startup times. Annoyingly, we now have to build redo using minimal/do, then run the tests using bin/redo. To make this less annoying, we add a toplevel ./do script that knows the right steps, and a Makefile (whee!) for people who are used to typing 'make' and 'make test' and 'make clean'.
2018-12-03 21:39:15 -05:00
[p+'/../redo' for p in dirnames] +
Automatically select a good shell instead of relying on /bin/sh. This includes a fairly detailed test of various known shell bugs from the autoconf docs. The idea here is that if redo works on your system, you should be able to rely on a *good* shell to run your .do files; you shouldn't have to work around zillions of bugs like autoconf does.
2010-12-21 04:19:50 -08:00
dirnames)
seen = {}
dirs = []
for k in trynames:
vars_init.py: always add .../redo-sh to the PATH, even if it doesn't exist. Otherwise a fresh 'redo test' in the redo repo won't switch to using redo-sh/sh after redo-sh.do runs, and the tests aren't accurate.
2011-01-18 00:19:14 -08:00
if not seen.get(k):
Automatically select a good shell instead of relying on /bin/sh. This includes a fairly detailed test of various known shell bugs from the autoconf docs. The idea here is that if redo works on your system, you should be able to rely on a *good* shell to run your .do files; you shouldn't have to work around zillions of bugs like autoconf does.
2010-12-21 04:19:50 -08:00
seen[k] = 1
dirs.append(k)
os.environ['PATH'] = ':'.join(dirs) + ':' + os.environ['PATH']
redo-ifchange can now be run even if there's no parent redo.
2010-12-11 19:08:53 -08:00
os.environ['REDO'] = os.path.abspath(sys.argv[0])
if not os.environ.get('REDO_BASE'):
base = os.path.commonprefix([os.path.abspath(os.path.dirname(t))
for t in targets] + [os.getcwd()])
bsplit = base.split('/')
for i in range(len(bsplit)-1, 0, -1):
newbase = '/'.join(bsplit[:i])
if os.path.exists(newbase + '/.redo'):
base = newbase
break
os.environ['REDO_BASE'] = base
os.environ['REDO_STARTDIR'] = os.getcwd()
import state
state.init()
Add cyclic dependence detection. If a depends on b which depends on a, redo would just freeze. Now it aborts with a somewhat helpful error message. [Updated by apenwarr for coding style and to add a test.]
2016-11-27 23:35:28 -08:00
os.environ['REDO_LOCKS'] = os.environ.get('REDO_LOCKS', '')
Reference in a new issue Copy permalink