2018-11-14 10:52:04 -08:00
|
|
|
#!/usr/bin/env python2
|
2010-11-21 02:08:05 -08:00
|
|
|
import sys, os
|
2010-12-11 18:32:40 -08:00
|
|
|
import options
|
|
|
|
|
from helpers import atoi
|
2010-11-12 05:24:46 -08:00
|
|
|
|
|
|
|
|
optspec = """
|
|
|
|
|
redo [targets...]
|
|
|
|
|
--
|
2010-11-13 04:36:44 -08:00
|
|
|
j,jobs= maximum number of jobs to build at once
|
2010-11-12 07:03:06 -08:00
|
|
|
d,debug print dependency checks as they happen
|
2010-11-21 06:35:52 -08:00
|
|
|
v,verbose print commands as they are read from .do files (variables intact)
|
|
|
|
|
x,xtrace print commands as they are executed (variables expanded)
|
2010-11-21 07:10:48 -08:00
|
|
|
k,keep-going keep going as long as possible even if some targets fail
|
2010-11-16 00:14:57 -08:00
|
|
|
shuffle randomize the build order to find dependency bugs
|
redo-log: capture and linearize the output of redo builds.
redo now saves the stderr from every .do script, for every target, into
a file in the .redo directory. That means you can look up the logs
from the most recent build of any target using the new redo-log
command, for example:
redo-log -r all
The default is to show logs non-recursively, that is, it'll show when a
target does redo-ifchange on another target, but it won't recurse into
the logs for the latter target. With -r (recursive), it does. With -u
(unchanged), it does even if redo-ifchange discovered that the target
was already up-to-date; in that case, it prints the logs of the *most
recent* time the target was generated.
With --no-details, redo-log will show only the 'redo' lines, not the
other log messages. For very noisy build systems (like recursing into
a 'make' instance) this can be helpful to get an overview of what
happened, without all the cruft.
You can use the -f (follow) option like tail -f, to follow a build
that's currently in progress until it finishes. redo itself spins up a
copy of redo-log -r -f while it runs, so you can see what's going on.
Still broken in this version:
- No man page or new tests yet.
- ANSI colors don't yet work (unless you use --raw-logs, which gives
the old-style behaviour).
- You can't redirect the output of a sub-redo to a file or a
pipe right now, because redo-log is eating it.
- The regex for matching 'redo' lines in the log is very gross.
Instead, we should put the raw log files in a more machine-parseable
format, and redo-log should turn that into human-readable format.
- redo-log tries to "linearize" the logs, which makes them
comprehensible even for a large parallel build. It recursively shows
log messages for each target in depth-first tree order (by tracing
into a new target every time it sees a 'redo' line). This works
really well, but in some specific cases, the "topmost" redo instance
can get stuck waiting for a jwack token, which makes it look like the
whole build has stalled, when really redo-log is just waiting a long
time for a particular subprocess to be able to continue. We'll need to
add a specific workaround for that.
2018-11-03 22:09:18 -04:00
|
|
|
no-details only show 'redo' recursion trace (to see more later, use redo-log)
|
|
|
|
|
no-status don't display build summary line at the bottom of the screen
|
|
|
|
|
raw-logs don't use redo-log, just send all output straight to stderr
|
2010-11-21 06:23:41 -08:00
|
|
|
debug-locks print messages about file locking (useful for debugging)
|
2010-11-22 01:50:46 -08:00
|
|
|
debug-pids print process ids as part of log messages (useful for debugging)
|
2011-01-25 17:13:35 -08:00
|
|
|
version print the current version and exit
|
2010-11-12 05:24:46 -08:00
|
|
|
"""
|
2011-12-21 21:33:22 -05:00
|
|
|
o = options.Options(optspec)
|
2010-11-12 05:24:46 -08:00
|
|
|
(opt, flags, extra) = o.parse(sys.argv[1:])
|
|
|
|
|
|
2011-03-10 21:10:15 -08:00
|
|
|
targets = extra
|
2010-11-12 05:24:46 -08:00
|
|
|
|
2011-01-25 17:13:35 -08:00
|
|
|
if opt.version:
|
|
|
|
|
import version
|
|
|
|
|
print version.TAG
|
|
|
|
|
sys.exit(0)
|
2010-11-13 00:53:55 -08:00
|
|
|
if opt.debug:
|
2010-11-16 04:13:17 -08:00
|
|
|
os.environ['REDO_DEBUG'] = str(opt.debug or 0)
|
2010-11-13 00:53:55 -08:00
|
|
|
if opt.verbose:
|
|
|
|
|
os.environ['REDO_VERBOSE'] = '1'
|
2010-11-21 06:35:52 -08:00
|
|
|
if opt.xtrace:
|
|
|
|
|
os.environ['REDO_XTRACE'] = '1'
|
2010-11-21 07:10:48 -08:00
|
|
|
if opt.keep_going:
|
|
|
|
|
os.environ['REDO_KEEP_GOING'] = '1'
|
2010-11-16 00:14:57 -08:00
|
|
|
if opt.shuffle:
|
|
|
|
|
os.environ['REDO_SHUFFLE'] = '1'
|
redo-log: capture and linearize the output of redo builds.
redo now saves the stderr from every .do script, for every target, into
a file in the .redo directory. That means you can look up the logs
from the most recent build of any target using the new redo-log
command, for example:
redo-log -r all
The default is to show logs non-recursively, that is, it'll show when a
target does redo-ifchange on another target, but it won't recurse into
the logs for the latter target. With -r (recursive), it does. With -u
(unchanged), it does even if redo-ifchange discovered that the target
was already up-to-date; in that case, it prints the logs of the *most
recent* time the target was generated.
With --no-details, redo-log will show only the 'redo' lines, not the
other log messages. For very noisy build systems (like recursing into
a 'make' instance) this can be helpful to get an overview of what
happened, without all the cruft.
You can use the -f (follow) option like tail -f, to follow a build
that's currently in progress until it finishes. redo itself spins up a
copy of redo-log -r -f while it runs, so you can see what's going on.
Still broken in this version:
- No man page or new tests yet.
- ANSI colors don't yet work (unless you use --raw-logs, which gives
the old-style behaviour).
- You can't redirect the output of a sub-redo to a file or a
pipe right now, because redo-log is eating it.
- The regex for matching 'redo' lines in the log is very gross.
Instead, we should put the raw log files in a more machine-parseable
format, and redo-log should turn that into human-readable format.
- redo-log tries to "linearize" the logs, which makes them
comprehensible even for a large parallel build. It recursively shows
log messages for each target in depth-first tree order (by tracing
into a new target every time it sees a 'redo' line). This works
really well, but in some specific cases, the "topmost" redo instance
can get stuck waiting for a jwack token, which makes it look like the
whole build has stalled, when really redo-log is just waiting a long
time for a particular subprocess to be able to continue. We'll need to
add a specific workaround for that.
2018-11-03 22:09:18 -04:00
|
|
|
if opt.raw_logs:
|
|
|
|
|
os.environ['REDO_RAW_LOGS'] = '1'
|
|
|
|
|
if opt.debug_locks or not os.environ.get('REDO_RAW_LOGS'):
|
|
|
|
|
# FIXME: force-enabled for redo-log, for now
|
2010-11-21 06:23:41 -08:00
|
|
|
os.environ['REDO_DEBUG_LOCKS'] = '1'
|
2010-11-22 01:50:46 -08:00
|
|
|
if opt.debug_pids:
|
|
|
|
|
os.environ['REDO_DEBUG_PIDS'] = '1'
|
2010-11-13 00:53:55 -08:00
|
|
|
|
2010-12-11 19:08:53 -08:00
|
|
|
import vars_init
|
|
|
|
|
vars_init.init(targets)
|
2010-11-19 03:13:40 -08:00
|
|
|
|
2010-12-11 18:32:40 -08:00
|
|
|
import vars, state, builder, jwack
|
Raw logs contain @@REDO lines instead of formatted data.
This makes them more reliable to parse. redo-log can parse each line,
format and print it, then recurse if necessary. This got a little ugly
because I wanted 'redo --raw-logs' to work, which we want to format the
output nicely, but not call redo-log.
(As a result, --raw-logs has a different meaning to redo and
redo-log, which is kinda dumb. I should fix that.)
As an added bonus, redo-log now handles indenting of recursive logs, so
if the build was a -> a/b -> a/b/c, and you look at the log for a/b, it
can still start at the top level indentation.
2018-11-13 04:05:42 -05:00
|
|
|
from logs import warn, err
|
2010-11-19 03:13:40 -08:00
|
|
|
|
2010-11-13 04:36:44 -08:00
|
|
|
try:
|
redo-log: capture and linearize the output of redo builds.
redo now saves the stderr from every .do script, for every target, into
a file in the .redo directory. That means you can look up the logs
from the most recent build of any target using the new redo-log
command, for example:
redo-log -r all
The default is to show logs non-recursively, that is, it'll show when a
target does redo-ifchange on another target, but it won't recurse into
the logs for the latter target. With -r (recursive), it does. With -u
(unchanged), it does even if redo-ifchange discovered that the target
was already up-to-date; in that case, it prints the logs of the *most
recent* time the target was generated.
With --no-details, redo-log will show only the 'redo' lines, not the
other log messages. For very noisy build systems (like recursing into
a 'make' instance) this can be helpful to get an overview of what
happened, without all the cruft.
You can use the -f (follow) option like tail -f, to follow a build
that's currently in progress until it finishes. redo itself spins up a
copy of redo-log -r -f while it runs, so you can see what's going on.
Still broken in this version:
- No man page or new tests yet.
- ANSI colors don't yet work (unless you use --raw-logs, which gives
the old-style behaviour).
- You can't redirect the output of a sub-redo to a file or a
pipe right now, because redo-log is eating it.
- The regex for matching 'redo' lines in the log is very gross.
Instead, we should put the raw log files in a more machine-parseable
format, and redo-log should turn that into human-readable format.
- redo-log tries to "linearize" the logs, which makes them
comprehensible even for a large parallel build. It recursively shows
log messages for each target in depth-first tree order (by tracing
into a new target every time it sees a 'redo' line). This works
really well, but in some specific cases, the "topmost" redo instance
can get stuck waiting for a jwack token, which makes it look like the
whole build has stalled, when really redo-log is just waiting a long
time for a particular subprocess to be able to continue. We'll need to
add a specific workaround for that.
2018-11-03 22:09:18 -04:00
|
|
|
if vars_init.is_toplevel:
|
|
|
|
|
builder.start_stdin_log_reader(status=opt.status, details=opt.details)
|
2010-12-11 21:19:15 -08:00
|
|
|
for t in targets:
|
|
|
|
|
if os.path.exists(t):
|
|
|
|
|
f = state.File(name=t)
|
|
|
|
|
if not f.is_generated:
|
|
|
|
|
warn('%s: exists and not marked as generated; not redoing.\n'
|
|
|
|
|
% f.nicename())
|
2018-10-11 02:51:12 -04:00
|
|
|
state.rollback()
|
2010-12-11 21:19:15 -08:00
|
|
|
|
2010-12-11 18:32:40 -08:00
|
|
|
j = atoi(opt.jobs or 1)
|
2010-11-13 04:36:44 -08:00
|
|
|
if j < 1 or j > 1000:
|
|
|
|
|
err('invalid --jobs value: %r\n' % opt.jobs)
|
|
|
|
|
jwack.setup(j)
|
|
|
|
|
try:
|
2018-10-06 04:36:24 -04:00
|
|
|
assert(state.is_flushed())
|
redo-log: capture and linearize the output of redo builds.
redo now saves the stderr from every .do script, for every target, into
a file in the .redo directory. That means you can look up the logs
from the most recent build of any target using the new redo-log
command, for example:
redo-log -r all
The default is to show logs non-recursively, that is, it'll show when a
target does redo-ifchange on another target, but it won't recurse into
the logs for the latter target. With -r (recursive), it does. With -u
(unchanged), it does even if redo-ifchange discovered that the target
was already up-to-date; in that case, it prints the logs of the *most
recent* time the target was generated.
With --no-details, redo-log will show only the 'redo' lines, not the
other log messages. For very noisy build systems (like recursing into
a 'make' instance) this can be helpful to get an overview of what
happened, without all the cruft.
You can use the -f (follow) option like tail -f, to follow a build
that's currently in progress until it finishes. redo itself spins up a
copy of redo-log -r -f while it runs, so you can see what's going on.
Still broken in this version:
- No man page or new tests yet.
- ANSI colors don't yet work (unless you use --raw-logs, which gives
the old-style behaviour).
- You can't redirect the output of a sub-redo to a file or a
pipe right now, because redo-log is eating it.
- The regex for matching 'redo' lines in the log is very gross.
Instead, we should put the raw log files in a more machine-parseable
format, and redo-log should turn that into human-readable format.
- redo-log tries to "linearize" the logs, which makes them
comprehensible even for a large parallel build. It recursively shows
log messages for each target in depth-first tree order (by tracing
into a new target every time it sees a 'redo' line). This works
really well, but in some specific cases, the "topmost" redo instance
can get stuck waiting for a jwack token, which makes it look like the
whole build has stalled, when really redo-log is just waiting a long
time for a particular subprocess to be able to continue. We'll need to
add a specific workaround for that.
2018-11-03 22:09:18 -04:00
|
|
|
retcode = builder.main(targets, lambda t: (True, True))
|
2018-10-06 04:36:24 -04:00
|
|
|
assert(state.is_flushed())
|
2010-11-13 04:36:44 -08:00
|
|
|
finally:
|
2018-10-06 04:36:24 -04:00
|
|
|
try:
|
|
|
|
|
state.rollback()
|
|
|
|
|
finally:
|
|
|
|
|
jwack.force_return_tokens()
|
redo-log: capture and linearize the output of redo builds.
redo now saves the stderr from every .do script, for every target, into
a file in the .redo directory. That means you can look up the logs
from the most recent build of any target using the new redo-log
command, for example:
redo-log -r all
The default is to show logs non-recursively, that is, it'll show when a
target does redo-ifchange on another target, but it won't recurse into
the logs for the latter target. With -r (recursive), it does. With -u
(unchanged), it does even if redo-ifchange discovered that the target
was already up-to-date; in that case, it prints the logs of the *most
recent* time the target was generated.
With --no-details, redo-log will show only the 'redo' lines, not the
other log messages. For very noisy build systems (like recursing into
a 'make' instance) this can be helpful to get an overview of what
happened, without all the cruft.
You can use the -f (follow) option like tail -f, to follow a build
that's currently in progress until it finishes. redo itself spins up a
copy of redo-log -r -f while it runs, so you can see what's going on.
Still broken in this version:
- No man page or new tests yet.
- ANSI colors don't yet work (unless you use --raw-logs, which gives
the old-style behaviour).
- You can't redirect the output of a sub-redo to a file or a
pipe right now, because redo-log is eating it.
- The regex for matching 'redo' lines in the log is very gross.
Instead, we should put the raw log files in a more machine-parseable
format, and redo-log should turn that into human-readable format.
- redo-log tries to "linearize" the logs, which makes them
comprehensible even for a large parallel build. It recursively shows
log messages for each target in depth-first tree order (by tracing
into a new target every time it sees a 'redo' line). This works
really well, but in some specific cases, the "topmost" redo instance
can get stuck waiting for a jwack token, which makes it look like the
whole build has stalled, when really redo-log is just waiting a long
time for a particular subprocess to be able to continue. We'll need to
add a specific workaround for that.
2018-11-03 22:09:18 -04:00
|
|
|
if vars_init.is_toplevel:
|
|
|
|
|
builder.await_log_reader()
|
2010-11-13 02:09:42 -08:00
|
|
|
sys.exit(retcode)
|
|
|
|
|
except KeyboardInterrupt:
|
redo-log: capture and linearize the output of redo builds.
redo now saves the stderr from every .do script, for every target, into
a file in the .redo directory. That means you can look up the logs
from the most recent build of any target using the new redo-log
command, for example:
redo-log -r all
The default is to show logs non-recursively, that is, it'll show when a
target does redo-ifchange on another target, but it won't recurse into
the logs for the latter target. With -r (recursive), it does. With -u
(unchanged), it does even if redo-ifchange discovered that the target
was already up-to-date; in that case, it prints the logs of the *most
recent* time the target was generated.
With --no-details, redo-log will show only the 'redo' lines, not the
other log messages. For very noisy build systems (like recursing into
a 'make' instance) this can be helpful to get an overview of what
happened, without all the cruft.
You can use the -f (follow) option like tail -f, to follow a build
that's currently in progress until it finishes. redo itself spins up a
copy of redo-log -r -f while it runs, so you can see what's going on.
Still broken in this version:
- No man page or new tests yet.
- ANSI colors don't yet work (unless you use --raw-logs, which gives
the old-style behaviour).
- You can't redirect the output of a sub-redo to a file or a
pipe right now, because redo-log is eating it.
- The regex for matching 'redo' lines in the log is very gross.
Instead, we should put the raw log files in a more machine-parseable
format, and redo-log should turn that into human-readable format.
- redo-log tries to "linearize" the logs, which makes them
comprehensible even for a large parallel build. It recursively shows
log messages for each target in depth-first tree order (by tracing
into a new target every time it sees a 'redo' line). This works
really well, but in some specific cases, the "topmost" redo instance
can get stuck waiting for a jwack token, which makes it look like the
whole build has stalled, when really redo-log is just waiting a long
time for a particular subprocess to be able to continue. We'll need to
add a specific workaround for that.
2018-11-03 22:09:18 -04:00
|
|
|
if vars_init.is_toplevel:
|
|
|
|
|
builder.await_log_reader()
|
2010-11-13 02:09:42 -08:00
|
|
|
sys.exit(200)
|