2018-11-26 17:04:31 -05:00
|
|
|
#
|
|
|
|
|
# Copyright 2010-2018 Avery Pennarun and contributors
|
2018-12-02 23:15:37 -05:00
|
|
|
#
|
2018-11-26 17:04:31 -05:00
|
|
|
# Licensed under the Apache License, Version 2.0 (the "License");
|
|
|
|
|
# you may not use this file except in compliance with the License.
|
|
|
|
|
# You may obtain a copy of the License at
|
2018-12-02 23:15:37 -05:00
|
|
|
#
|
2018-11-26 17:04:31 -05:00
|
|
|
# http://www.apache.org/licenses/LICENSE-2.0
|
2018-12-02 23:15:37 -05:00
|
|
|
#
|
2018-11-26 17:04:31 -05:00
|
|
|
# Unless required by applicable law or agreed to in writing, software
|
|
|
|
|
# distributed under the License is distributed on an "AS IS" BASIS,
|
|
|
|
|
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
|
|
|
|
# See the License for the specific language governing permissions and
|
|
|
|
|
# limitations under the License.
|
|
|
|
|
#
|
redo-log: prioritize the "foreground" process.
When running a parallel build, redo-log -f (which is auto-started by
redo) tries to traverse through the logs depth first, in the order
parent processes started subprocesses. This works pretty well, but if
its dependencies are locked, a process might have to give up its
jobserver token while other stuff builds its dependencies. After the
dependency finishes, the parent might not be able to get a token for
quite some time, and the logs will appear to stop.
To prevent this from happening, we can instantiate up to one "cheater"
token, only in the foreground process (the one locked by redo-log -f),
which will allow it to continue running, albeit a bit slowly (since it
only has one token out of possibly many). When the process finishes,
we then destroy the fake token. It gets a little complicated; see
explanation at the top of jwack.py.
2018-11-17 04:32:09 -05:00
|
|
|
import sys, os, traceback
|
2010-12-11 18:32:40 -08:00
|
|
|
import options
|
2018-12-02 23:15:37 -05:00
|
|
|
from atoi import atoi
|
2010-11-12 05:24:46 -08:00
|
|
|
|
|
|
|
|
optspec = """
|
|
|
|
|
redo [targets...]
|
|
|
|
|
--
|
2010-11-13 04:36:44 -08:00
|
|
|
j,jobs= maximum number of jobs to build at once
|
2010-11-12 07:03:06 -08:00
|
|
|
d,debug print dependency checks as they happen
|
2010-11-21 06:35:52 -08:00
|
|
|
v,verbose print commands as they are read from .do files (variables intact)
|
|
|
|
|
x,xtrace print commands as they are executed (variables expanded)
|
2010-11-21 07:10:48 -08:00
|
|
|
k,keep-going keep going as long as possible even if some targets fail
|
2010-11-16 00:14:57 -08:00
|
|
|
shuffle randomize the build order to find dependency bugs
|
2018-12-02 22:35:57 -05:00
|
|
|
version print the current version and exit
|
|
|
|
|
|
|
|
|
|
redo-log options:
|
|
|
|
|
no-log don't capture error output, just let it flow straight to stderr
|
redo-log: capture and linearize the output of redo builds.
redo now saves the stderr from every .do script, for every target, into
a file in the .redo directory. That means you can look up the logs
from the most recent build of any target using the new redo-log
command, for example:
redo-log -r all
The default is to show logs non-recursively, that is, it'll show when a
target does redo-ifchange on another target, but it won't recurse into
the logs for the latter target. With -r (recursive), it does. With -u
(unchanged), it does even if redo-ifchange discovered that the target
was already up-to-date; in that case, it prints the logs of the *most
recent* time the target was generated.
With --no-details, redo-log will show only the 'redo' lines, not the
other log messages. For very noisy build systems (like recursing into
a 'make' instance) this can be helpful to get an overview of what
happened, without all the cruft.
You can use the -f (follow) option like tail -f, to follow a build
that's currently in progress until it finishes. redo itself spins up a
copy of redo-log -r -f while it runs, so you can see what's going on.
Still broken in this version:
- No man page or new tests yet.
- ANSI colors don't yet work (unless you use --raw-logs, which gives
the old-style behaviour).
- You can't redirect the output of a sub-redo to a file or a
pipe right now, because redo-log is eating it.
- The regex for matching 'redo' lines in the log is very gross.
Instead, we should put the raw log files in a more machine-parseable
format, and redo-log should turn that into human-readable format.
- redo-log tries to "linearize" the logs, which makes them
comprehensible even for a large parallel build. It recursively shows
log messages for each target in depth-first tree order (by tracing
into a new target every time it sees a 'redo' line). This works
really well, but in some specific cases, the "topmost" redo instance
can get stuck waiting for a jwack token, which makes it look like the
whole build has stalled, when really redo-log is just waiting a long
time for a particular subprocess to be able to continue. We'll need to
add a specific workaround for that.
2018-11-03 22:09:18 -04:00
|
|
|
no-details only show 'redo' recursion trace (to see more later, use redo-log)
|
|
|
|
|
no-status don't display build summary line at the bottom of the screen
|
2018-11-19 10:55:56 -05:00
|
|
|
no-pretty don't pretty-print logs, show raw @@REDO output instead
|
2018-11-19 11:22:53 -05:00
|
|
|
no-color disable ANSI color; --color to force enable (default: auto)
|
2010-11-21 06:23:41 -08:00
|
|
|
debug-locks print messages about file locking (useful for debugging)
|
2010-11-22 01:50:46 -08:00
|
|
|
debug-pids print process ids as part of log messages (useful for debugging)
|
2010-11-12 05:24:46 -08:00
|
|
|
"""
|
2011-12-21 21:33:22 -05:00
|
|
|
o = options.Options(optspec)
|
2010-11-12 05:24:46 -08:00
|
|
|
(opt, flags, extra) = o.parse(sys.argv[1:])
|
|
|
|
|
|
2011-03-10 21:10:15 -08:00
|
|
|
targets = extra
|
2010-11-12 05:24:46 -08:00
|
|
|
|
2011-01-25 17:13:35 -08:00
|
|
|
if opt.version:
|
|
|
|
|
import version
|
|
|
|
|
print version.TAG
|
|
|
|
|
sys.exit(0)
|
2010-11-13 00:53:55 -08:00
|
|
|
if opt.debug:
|
2010-11-16 04:13:17 -08:00
|
|
|
os.environ['REDO_DEBUG'] = str(opt.debug or 0)
|
2010-11-13 00:53:55 -08:00
|
|
|
if opt.verbose:
|
|
|
|
|
os.environ['REDO_VERBOSE'] = '1'
|
2010-11-21 06:35:52 -08:00
|
|
|
if opt.xtrace:
|
|
|
|
|
os.environ['REDO_XTRACE'] = '1'
|
2010-11-21 07:10:48 -08:00
|
|
|
if opt.keep_going:
|
|
|
|
|
os.environ['REDO_KEEP_GOING'] = '1'
|
2010-11-16 00:14:57 -08:00
|
|
|
if opt.shuffle:
|
|
|
|
|
os.environ['REDO_SHUFFLE'] = '1'
|
2018-11-19 10:55:56 -05:00
|
|
|
if opt.debug_locks:
|
2010-11-21 06:23:41 -08:00
|
|
|
os.environ['REDO_DEBUG_LOCKS'] = '1'
|
2010-11-22 01:50:46 -08:00
|
|
|
if opt.debug_pids:
|
|
|
|
|
os.environ['REDO_DEBUG_PIDS'] = '1'
|
2010-11-13 00:53:55 -08:00
|
|
|
|
2018-11-19 10:55:56 -05:00
|
|
|
# This is slightly tricky: the log and pretty options default to true. We
|
|
|
|
|
# want to inherit that 'true' value from parent processes *unless* someone
|
|
|
|
|
# explicitly specifies the reverse.
|
|
|
|
|
if opt.no_log:
|
|
|
|
|
os.environ['REDO_LOG'] = '0'
|
|
|
|
|
if opt.no_pretty:
|
|
|
|
|
os.environ['REDO_PRETTY'] = '0'
|
2018-11-19 11:22:53 -05:00
|
|
|
if opt.no_color:
|
|
|
|
|
os.environ['REDO_COLOR'] = '0'
|
2018-11-19 10:55:56 -05:00
|
|
|
|
2010-12-11 19:08:53 -08:00
|
|
|
import vars_init
|
|
|
|
|
vars_init.init(targets)
|
2010-11-19 03:13:40 -08:00
|
|
|
|
2010-12-11 18:32:40 -08:00
|
|
|
import vars, state, builder, jwack
|
Raw logs contain @@REDO lines instead of formatted data.
This makes them more reliable to parse. redo-log can parse each line,
format and print it, then recurse if necessary. This got a little ugly
because I wanted 'redo --raw-logs' to work, which we want to format the
output nicely, but not call redo-log.
(As a result, --raw-logs has a different meaning to redo and
redo-log, which is kinda dumb. I should fix that.)
As an added bonus, redo-log now handles indenting of recursive logs, so
if the build was a -> a/b -> a/b/c, and you look at the log for a/b, it
can still start at the top level indentation.
2018-11-13 04:05:42 -05:00
|
|
|
from logs import warn, err
|
2010-11-19 03:13:40 -08:00
|
|
|
|
2018-12-03 21:15:31 -05:00
|
|
|
def main():
|
2010-11-13 04:36:44 -08:00
|
|
|
try:
|
2018-12-03 21:15:31 -05:00
|
|
|
j = atoi(opt.jobs or 1)
|
|
|
|
|
if vars_init.is_toplevel and (vars.LOG or j > 1):
|
|
|
|
|
builder.close_stdin()
|
|
|
|
|
if vars_init.is_toplevel and vars.LOG:
|
|
|
|
|
builder.start_stdin_log_reader(
|
|
|
|
|
status=opt.status, details=opt.details,
|
|
|
|
|
pretty=opt.pretty, color=opt.color,
|
|
|
|
|
debug_locks=opt.debug_locks, debug_pids=opt.debug_pids)
|
|
|
|
|
for t in targets:
|
|
|
|
|
if os.path.exists(t):
|
|
|
|
|
f = state.File(name=t)
|
|
|
|
|
if not f.is_generated:
|
|
|
|
|
warn('%s: exists and not marked as generated; not redoing.\n'
|
|
|
|
|
% f.nicename())
|
|
|
|
|
state.rollback()
|
|
|
|
|
|
|
|
|
|
if j < 1 or j > 1000:
|
|
|
|
|
err('invalid --jobs value: %r\n' % opt.jobs)
|
|
|
|
|
jwack.setup(j)
|
2018-10-06 04:36:24 -04:00
|
|
|
try:
|
2018-12-03 21:15:31 -05:00
|
|
|
assert state.is_flushed()
|
|
|
|
|
retcode = builder.main(targets, lambda t: (True, True))
|
|
|
|
|
assert state.is_flushed()
|
2018-10-06 04:36:24 -04:00
|
|
|
finally:
|
redo-log: prioritize the "foreground" process.
When running a parallel build, redo-log -f (which is auto-started by
redo) tries to traverse through the logs depth first, in the order
parent processes started subprocesses. This works pretty well, but if
its dependencies are locked, a process might have to give up its
jobserver token while other stuff builds its dependencies. After the
dependency finishes, the parent might not be able to get a token for
quite some time, and the logs will appear to stop.
To prevent this from happening, we can instantiate up to one "cheater"
token, only in the foreground process (the one locked by redo-log -f),
which will allow it to continue running, albeit a bit slowly (since it
only has one token out of possibly many). When the process finishes,
we then destroy the fake token. It gets a little complicated; see
explanation at the top of jwack.py.
2018-11-17 04:32:09 -05:00
|
|
|
try:
|
2018-12-03 21:15:31 -05:00
|
|
|
state.rollback()
|
|
|
|
|
finally:
|
|
|
|
|
try:
|
|
|
|
|
jwack.force_return_tokens()
|
|
|
|
|
except Exception, e: # pylint: disable=broad-except
|
|
|
|
|
traceback.print_exc(100, sys.stderr)
|
|
|
|
|
err('unexpected error: %r\n' % e)
|
|
|
|
|
retcode = 1
|
|
|
|
|
if vars_init.is_toplevel:
|
|
|
|
|
builder.await_log_reader()
|
|
|
|
|
sys.exit(retcode)
|
|
|
|
|
except KeyboardInterrupt:
|
|
|
|
|
if vars_init.is_toplevel:
|
|
|
|
|
builder.await_log_reader()
|
|
|
|
|
sys.exit(200)
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
if __name__ == '__main__':
|
|
|
|
|
main()
|