forks/apenwarr-redo
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
apenwarr-redo/vars.py

42 lines
1.5 KiB
Python
Raw Normal View History

Use a single .redo dir for the entire build.
2010-11-12 22:47:03 -08:00
import os
The only thing in helpers.py that needed vars.py was the log stuff. So put it in its own file. Now it's safer to import and use helpers even if you can't safely touch vars.
2010-12-11 18:32:40 -08:00
from atoi import atoi
Use a single .redo dir for the entire build.
2010-11-12 22:47:03 -08:00
redo-if{change,create}: print a useful message if REDO isn't set. Again, I forgot to make vars.py not crash if the variables aren't set, so we can print a useful error message. But this time I have the right solution: vars.py will do the checking for itself, and abort with a nice message.
2010-11-22 02:45:00 -08:00
if not os.environ.get('REDO'):
import sys
sys.stderr.write('%s: error: must be run from inside a .do\n'
% sys.argv[0])
sys.exit(100)
Remove the need for relpath (and thus abspath) in builder.py.
2010-11-21 03:34:32 -08:00
PWD = os.environ.get('REDO_PWD', '')
Move log stuff into a separate log.py.
2010-11-13 00:11:34 -08:00
TARGET = os.environ.get('REDO_TARGET', '')
DEPTH = os.environ.get('REDO_DEPTH', '')
The only thing in helpers.py that needed vars.py was the log stuff. So put it in its own file. Now it's safer to import and use helpers even if you can't safely touch vars.
2010-12-11 18:32:40 -08:00
DEBUG = atoi(os.environ.get('REDO_DEBUG', ''))
Add a --debug-locks option. Get rid of the "locked..." and "...unlocked!" messages by default, since they're not usually interesting. But add a new option to bring them back in case we end up with trouble debugging the locking stuff. (I don't really 100% trust it yet, although I haven't had a problem for a while now.)
2010-11-21 06:23:41 -08:00
DEBUG_LOCKS = os.environ.get('REDO_DEBUG_LOCKS', '') and 1 or 0
Add a new --debug-pids option. This makes the helpers.* log functions prepend getpid() to each line, so you can see which pid did what.
2010-11-22 01:50:46 -08:00
DEBUG_PIDS = os.environ.get('REDO_DEBUG_PIDS', '') and 1 or 0
Move log stuff into a separate log.py.
2010-11-13 00:11:34 -08:00
VERBOSE = os.environ.get('REDO_VERBOSE', '') and 1 or 0
Add a -x option that just passes -x to the subshell. This is often more useful than -v, since it prints the actual commands being executed, not just the lines being input from the script.
2010-11-21 06:35:52 -08:00
XTRACE = os.environ.get('REDO_XTRACE', '') and 1 or 0
Add a new -k (--keep-going) option, like make has. Previously, the default was to *always* keep going, which is actually not usually what you want. Now we actually exit correctly after an error. Of course you still might have multiple errors before existing if you were building in parallel.
2010-11-21 07:10:48 -08:00
KEEP_GOING = os.environ.get('REDO_KEEP_GOING', '') and 1 or 0
redo-log: capture and linearize the output of redo builds. redo now saves the stderr from every .do script, for every target, into a file in the .redo directory. That means you can look up the logs from the most recent build of any target using the new redo-log command, for example: redo-log -r all The default is to show logs non-recursively, that is, it'll show when a target does redo-ifchange on another target, but it won't recurse into the logs for the latter target. With -r (recursive), it does. With -u (unchanged), it does even if redo-ifchange discovered that the target was already up-to-date; in that case, it prints the logs of the *most recent* time the target was generated. With --no-details, redo-log will show only the 'redo' lines, not the other log messages. For very noisy build systems (like recursing into a 'make' instance) this can be helpful to get an overview of what happened, without all the cruft. You can use the -f (follow) option like tail -f, to follow a build that's currently in progress until it finishes. redo itself spins up a copy of redo-log -r -f while it runs, so you can see what's going on. Still broken in this version: - No man page or new tests yet. - ANSI colors don't yet work (unless you use --raw-logs, which gives the old-style behaviour). - You can't redirect the output of a sub-redo to a file or a pipe right now, because redo-log is eating it. - The regex for matching 'redo' lines in the log is very gross. Instead, we should put the raw log files in a more machine-parseable format, and redo-log should turn that into human-readable format. - redo-log tries to "linearize" the logs, which makes them comprehensible even for a large parallel build. It recursively shows log messages for each target in depth-first tree order (by tracing into a new target every time it sees a 'redo' line). This works really well, but in some specific cases, the "topmost" redo instance can get stuck waiting for a jwack token, which makes it look like the whole build has stalled, when really redo-log is just waiting a long time for a particular subprocess to be able to continue. We'll need to add a specific workaround for that.
2018-11-03 22:09:18 -04:00
RAW_LOGS = os.environ.get('REDO_RAW_LOGS', '') and 1 or 0
Add a --shuffle option to let you enable dependency randomization. Previously, for testing, we were *always* randomizing the build order of dependencies. That's annoying since it'll make build logs differ randomly from one run to the next, which could make comparisons harder. However, the feature is still useful for uncovering hidden dependencies between objects.
2010-11-16 00:14:57 -08:00
SHUFFLE = os.environ.get('REDO_SHUFFLE', '') and 1 or 0
redo-ifchange can now be run even if there's no parent redo.
2010-12-11 19:08:53 -08:00
STARTDIR = os.environ.get('REDO_STARTDIR', '')
The only thing in helpers.py that needed vars.py was the log stuff. So put it in its own file. Now it's safer to import and use helpers even if you can't safely touch vars.
2010-12-11 18:32:40 -08:00
RUNID = atoi(os.environ.get('REDO_RUNID')) or None
vars.py: it's safe to assume REDO_BASE and REDO_STARTDIR are set correctly. This lets us remove an extra call to os.abspath().
2010-11-21 03:09:21 -08:00
BASE = os.environ['REDO_BASE']
while BASE and BASE.endswith('/'):
BASE = BASE[:-1]
The second half of redo-stamp: out-of-order building. If a depends on b depends on c, and c is dirty but b uses redo-stamp checksums, then 'redo-ifchange a' is indeterminate: we won't know if we need to run a.do unless we first build b, but the script that *normally* runs 'redo-ifchange b' is a.do, and we don't want to run that yet, because we don't know for sure if b is dirty, and we shouldn't build a unless one of its dependencies is dirty. Eek! Luckily, there's a safe solution. If we *know* a is dirty - eg. because a.do or one of its children has definitely changed - then we can just run a.do immediately and there's no problem, even if b is indeterminate, because we were going to run a.do anyhow. If a's dependencies are *not* definitely dirty, and all we have is indeterminate ones like b, then that means a's build process *hasn't changed*, which means its tree of dependencies still includes b, which means we can deduce that if we *did* run a.do, it would end up running b.do. Since we know that anyhow, we can safely just run b.do, which will either b.set_checked() or b.set_changed(). Once that's done, we can re-parse a's dependencies and this time conclusively tell if it needs to be redone or not. Even if it does, b is already up-to-date, so the 'redo-ifchange b' line in a.do will be fast. ...now take all the above and do it recursively to handle nested dependencies, etc, and you're done.
2010-12-11 04:40:05 -08:00
UNLOCKED = os.environ.get('REDO_UNLOCKED', '') and 1 or 0
os.environ['REDO_UNLOCKED'] = '' # not inheritable by subprocesses
Whoops, redo-oob was slightly wrong when used with -j. We called 'redo' instead of 'redo-ifchange' on our indeterminate objects. Since other instances of redo-oob might be running at the same time, this could cause the same object to get rebuilt more than once unnecessarily. The unit tests caught this, I just didn't notice earlier.
2010-12-11 05:50:29 -08:00
NO_OOB = os.environ.get('REDO_NO_OOB', '') and 1 or 0
os.environ['REDO_NO_OOB'] = '' # not inheritable by subprocesses
Add cyclic dependence detection. If a depends on b which depends on a, redo would just freeze. Now it aborts with a somewhat helpful error message. [Updated by apenwarr for coding style and to add a test.]
2016-11-27 23:35:28 -08:00
def get_locks():
"""Get the list of held locks."""
return os.environ.get('REDO_LOCKS', '').split(':')
def add_lock(name):
"""Add a lock to the list of held locks."""
locks = set(get_locks())
locks.add(name)
os.environ['REDO_LOCKS'] = ':'.join(list(locks))
Reference in a new issue Copy permalink