redo-log: capture and linearize the output of redo builds.

redo now saves the stderr from every .do script, for every target, into
a file in the .redo directory.  That means you can look up the logs
from the most recent build of any target using the new redo-log
command, for example:

	redo-log -r all

The default is to show logs non-recursively, that is, it'll show when a
target does redo-ifchange on another target, but it won't recurse into
the logs for the latter target.  With -r (recursive), it does.  With -u
(unchanged), it does even if redo-ifchange discovered that the target
was already up-to-date; in that case, it prints the logs of the *most
recent* time the target was generated.

With --no-details, redo-log will show only the 'redo' lines, not the
other log messages.  For very noisy build systems (like recursing into
a 'make' instance) this can be helpful to get an overview of what
happened, without all the cruft.

You can use the -f (follow) option like tail -f, to follow a build
that's currently in progress until it finishes.  redo itself spins up a
copy of redo-log -r -f while it runs, so you can see what's going on.

Still broken in this version:

- No man page or new tests yet.

- ANSI colors don't yet work (unless you use --raw-logs, which gives
  the old-style behaviour).

- You can't redirect the output of a sub-redo to a file or a
  pipe right now, because redo-log is eating it.

- The regex for matching 'redo' lines in the log is very gross.
  Instead, we should put the raw log files in a more machine-parseable
  format, and redo-log should turn that into human-readable format.

- redo-log tries to "linearize" the logs, which makes them
  comprehensible even for a large parallel build.  It recursively shows
  log messages for each target in depth-first tree order (by tracing
  into a new target every time it sees a 'redo' line).  This works
  really well, but in some specific cases, the "topmost" redo instance
  can get stuck waiting for a jwack token, which makes it look like the
  whole build has stalled, when really redo-log is just waiting a long
  time for a particular subprocess to be able to continue.  We'll need to
  add a specific workaround for that.

redo.py

@@ -12,6 +12,9 @@ v,verbose  print commands as they are read from .do files (variables intact)
 x,xtrace   print commands as they are executed (variables expanded)
 k,keep-going  keep going as long as possible even if some targets fail
 shuffle    randomize the build order to find dependency bugs
+no-details only show 'redo' recursion trace (to see more later, use redo-log)
+no-status  don't display build summary line at the bottom of the screen
+raw-logs   don't use redo-log, just send all output straight to stderr
 debug-locks  print messages about file locking (useful for debugging)
 debug-pids   print process ids as part of log messages (useful for debugging)
 version    print the current version and exit
@@ -35,7 +38,10 @@ if opt.keep_going:
     os.environ['REDO_KEEP_GOING'] = '1'
 if opt.shuffle:
     os.environ['REDO_SHUFFLE'] = '1'
-if opt.debug_locks:
+if opt.raw_logs:
+    os.environ['REDO_RAW_LOGS'] = '1'
+if opt.debug_locks or not os.environ.get('REDO_RAW_LOGS'):
+    # FIXME: force-enabled for redo-log, for now
     os.environ['REDO_DEBUG_LOCKS'] = '1'
 if opt.debug_pids:
     os.environ['REDO_DEBUG_PIDS'] = '1'
@@ -47,6 +53,8 @@ import vars, state, builder, jwack
 from log import warn, err
 
 try:
+    if vars_init.is_toplevel:
+        builder.start_stdin_log_reader(status=opt.status, details=opt.details)
     for t in targets:
         if os.path.exists(t):
             f = state.File(name=t)
@@ -61,13 +69,17 @@ try:
     jwack.setup(j)
     try:
         assert(state.is_flushed())
-        retcode = builder.main(targets, lambda t: True)
+        retcode = builder.main(targets, lambda t: (True, True))
         assert(state.is_flushed())
     finally:
         try:
             state.rollback()
         finally:
             jwack.force_return_tokens()
+    if vars_init.is_toplevel:
+        builder.await_log_reader()
     sys.exit(retcode)
 except KeyboardInterrupt:
+    if vars_init.is_toplevel:
+        builder.await_log_reader()
     sys.exit(200)