forks/apenwarr-redo
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

The second half of redo-stamp: out-of-order building.

Browse source 
If a depends on b depends on c, and c is dirty but b uses redo-stamp
checksums, then 'redo-ifchange a' is indeterminate: we won't know if we need
to run a.do unless we first build b, but the script that *normally* runs
'redo-ifchange b' is a.do, and we don't want to run that yet, because we
don't know for sure if b is dirty, and we shouldn't build a unless one of
its dependencies is dirty.  Eek!

Luckily, there's a safe solution.  If we *know* a is dirty - eg. because
a.do or one of its children has definitely changed - then we can just run
a.do immediately and there's no problem, even if b is indeterminate, because
we were going to run a.do anyhow.

If a's dependencies are *not* definitely dirty, and all we have is
indeterminate ones like b, then that means a's build process *hasn't
changed*, which means its tree of dependencies still includes b, which means
we can deduce that if we *did* run a.do, it would end up running b.do.

Since we know that anyhow, we can safely just run b.do, which will either
b.set_checked() or b.set_changed().  Once that's done, we can re-parse a's
dependencies and this time conclusively tell if it needs to be redone or
not.  Even if it does, b is already up-to-date, so the 'redo-ifchange b'
line in a.do will be fast.

...now take all the above and do it recursively to handle nested
dependencies, etc, and you're done.
This commit is contained in:
Avery Pennarun 2010-12-11 04:40:05 -08:00
commit f702417ef3
6 changed files with 124 additions and 26 deletions
Download patch file Download diff file Expand all files Collapse all files

73
redo-ifchange.py
View file

@ -6,46 +6,82 @@ from helpers import debug, debug2, err, unlink
def _nice(t):
return state.relpath(os.path.join(vars.BASE, t), vars.STARTDIR)
CLEAN = 0
DIRTY = 1
def dirty_deps(f, depth, max_changed):
if vars.DEBUG >= 1:
debug('%s?%s\n' % (depth, _nice(f.name)))
if f.failed_runid:
debug('%s-- DIRTY (failed last time)\n' % depth)
return True
return DIRTY
if f.changed_runid == None:
debug('%s-- DIRTY (never built)\n' % depth)
return True
return DIRTY
if f.changed_runid > max_changed:
debug('%s-- DIRTY (built)\n' % depth)
return True # has been built more recently than parent
return DIRTY # has been built more recently than parent
if f.is_checked():
if vars.DEBUG >= 1: debug('%s-- CLEAN (checked)\n' % depth)
return False # has already been checked during this session
if vars.DEBUG >= 1:
debug('%s-- CLEAN (checked)\n' % depth)
return CLEAN # has already been checked during this session
if not f.stamp:
debug('%s-- DIRTY (no stamp)\n' % depth)
return True
return DIRTY
if f.stamp != f.read_stamp():
debug('%s-- DIRTY (mtime)\n' % depth)
return True
return DIRTY
must_build = []
for mode,f2 in f.deps():
dirty = CLEAN
if mode == 'c':
if os.path.exists(os.path.join(vars.BASE, f2.name)):
debug('%s-- DIRTY (created)\n' % depth)
return True
dirty = DIRTY
elif mode == 'm':
if dirty_deps(f2, depth = depth + ' ',
max_changed = max(f.changed_runid, f.checked_runid)):
sub = dirty_deps(f2, depth = depth + ' ',
max_changed = max(f.changed_runid,
f.checked_runid))
if sub:
debug('%s-- DIRTY (sub)\n' % depth)
return True
dirty = sub
else:
assert(mode in ('c','m'))
if not f.csum:
# f is a "normal" target: dirty f2 means f is instantly dirty
if dirty:
# if dirty==DIRTY, this means f is definitely dirty.
# if dirty==[...], it's a list of the uncertain children.
return dirty
else:
# f is "checksummable": dirty f2 means f needs to redo,
# but f might turn out to be clean after that (ie. our parent
# might not be dirty).
if dirty == DIRTY:
# f2 is definitely dirty, so f definitely needs to
# redo. However, after that, f might turn out to be
# unchanged.
return [f]
elif isinstance(dirty,list):
# our child f2 might be dirty, but it's not sure yet. It's
# given us a list of targets we have to redo in order to
# be sure.
must_build += dirty
if must_build:
# f is *maybe* dirty because at least one of its children is maybe
# dirty. must_build has accumulated a list of "topmost" uncertain
# objects in the tree. If we build all those, we can then
# redo-ifchange f and it won't have any uncertainty next time.
return must_build
# if we get here, it's because the target is clean
if f.is_override:
builder.warn_override(f.name)
f.set_checked()
f.save()
return False
return CLEAN
def should_build(t):
@ -63,9 +99,10 @@ try:
debug2('TARGET: %r %r %r\n' % (vars.STARTDIR, vars.PWD, vars.TARGET))
try:
targets = sys.argv[1:]
for t in targets:
f.add_dep('m', t)
f.save()
if not vars.UNLOCKED:
for t in targets:
f.add_dep('m', t)
f.save()
rv = builder.main(targets, should_build)
finally:
jwack.force_return_tokens()