redo-log: prioritize the "foreground" process.

When running a parallel build, redo-log -f (which is auto-started by
redo) tries to traverse through the logs depth first, in the order
parent processes started subprocesses.  This works pretty well, but if
its dependencies are locked, a process might have to give up its
jobserver token while other stuff builds its dependencies.  After the
dependency finishes, the parent might not be able to get a token for
quite some time, and the logs will appear to stop.

To prevent this from happening, we can instantiate up to one "cheater"
token, only in the foreground process (the one locked by redo-log -f),
which will allow it to continue running, albeit a bit slowly (since it
only has one token out of possibly many).  When the process finishes,
we then destroy the fake token.  It gets a little complicated; see
explanation at the top of jwack.py.
This commit is contained in:
Avery Pennarun 2018-11-17 04:32:09 -05:00
commit 8b5a567b2e
7 changed files with 348 additions and 104 deletions

11
logs.py
View file

@ -2,9 +2,9 @@ import os, re, sys, time
import vars
def check_tty():
def check_tty(file):
global RED, GREEN, YELLOW, BOLD, PLAIN
if sys.stderr.isatty() and (os.environ.get('TERM') or 'dumb') != 'dumb':
if file.isatty() and (os.environ.get('TERM') or 'dumb') != 'dumb':
# ...use ANSI formatting codes.
RED = "\x1b[31m"
GREEN = "\x1b[32m"
@ -52,15 +52,14 @@ class PrettyLog(object):
sys.stderr.flush()
g = REDO_RE.match(s)
if g:
# FIXME: support vars.DEBUG_PIDS somewhere
all = g.group(0)
self.file.write(s[:-len(all)])
words = g.group(1).split(':')
text = g.group(2)
kind, pid, when = words[0:3]
pid = int(pid)
if kind == 'unchanged':
if vars.DEBUG >= 1:
self._pretty(pid, '', '%s (unchanged)' % text)
self._pretty(pid, '', '%s (unchanged)' % text)
elif kind == 'check':
self._pretty(pid, GREEN, '(%s)' % text)
elif kind == 'do':
@ -102,7 +101,7 @@ _log = None
def setup(file, pretty):
global _log
if pretty:
check_tty()
check_tty(file)
_log = PrettyLog(file=file)
else:
_log = RawLog(file=file)