forks/apenwarr-redo
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
apenwarr-redo/redo-oob.py

30 lines
649 B
Python
Raw Normal View History

The second half of redo-stamp: out-of-order building. If a depends on b depends on c, and c is dirty but b uses redo-stamp checksums, then 'redo-ifchange a' is indeterminate: we won't know if we need to run a.do unless we first build b, but the script that *normally* runs 'redo-ifchange b' is a.do, and we don't want to run that yet, because we don't know for sure if b is dirty, and we shouldn't build a unless one of its dependencies is dirty. Eek! Luckily, there's a safe solution. If we *know* a is dirty - eg. because a.do or one of its children has definitely changed - then we can just run a.do immediately and there's no problem, even if b is indeterminate, because we were going to run a.do anyhow. If a's dependencies are *not* definitely dirty, and all we have is indeterminate ones like b, then that means a's build process *hasn't changed*, which means its tree of dependencies still includes b, which means we can deduce that if we *did* run a.do, it would end up running b.do. Since we know that anyhow, we can safely just run b.do, which will either b.set_checked() or b.set_changed(). Once that's done, we can re-parse a's dependencies and this time conclusively tell if it needs to be redone or not. Even if it does, b is already up-to-date, so the 'redo-ifchange b' line in a.do will be fast. ...now take all the above and do it recursively to handle nested dependencies, etc, and you're done.
2010-12-11 04:40:05 -08:00
#!/usr/bin/python
import sys, os
import state
from helpers import err
if len(sys.argv[1:]) < 2:
err('%s: at least 2 arguments expected.\n' % sys.argv[0])
sys.exit(1)
target = sys.argv[1]
deps = sys.argv[2:]
Fix a deadlock with redo-oob. If a checksummed target A used to exist but is now missing, and we tried to redo-ifchange that exact file, we would unnecessarily run 'redo-oob A A'; that is, we have to build A in order to determine if A needs to be built. The sub-targets of redo-oob aren't run with REDO_UNLOCKED, so this would deadlock instantly. Add an assertion to redo-oob to ensure we never try to redo-ifchange the primary target (thus converting the deadlock into an exception). And skip doing redo-oob when the target is already the same as the thing we have to check.
2010-12-11 06:16:32 -08:00
for d in deps:
assert(d != target)
The second half of redo-stamp: out-of-order building. If a depends on b depends on c, and c is dirty but b uses redo-stamp checksums, then 'redo-ifchange a' is indeterminate: we won't know if we need to run a.do unless we first build b, but the script that *normally* runs 'redo-ifchange b' is a.do, and we don't want to run that yet, because we don't know for sure if b is dirty, and we shouldn't build a unless one of its dependencies is dirty. Eek! Luckily, there's a safe solution. If we *know* a is dirty - eg. because a.do or one of its children has definitely changed - then we can just run a.do immediately and there's no problem, even if b is indeterminate, because we were going to run a.do anyhow. If a's dependencies are *not* definitely dirty, and all we have is indeterminate ones like b, then that means a's build process *hasn't changed*, which means its tree of dependencies still includes b, which means we can deduce that if we *did* run a.do, it would end up running b.do. Since we know that anyhow, we can safely just run b.do, which will either b.set_checked() or b.set_changed(). Once that's done, we can re-parse a's dependencies and this time conclusively tell if it needs to be redone or not. Even if it does, b is already up-to-date, so the 'redo-ifchange b' line in a.do will be fast. ...now take all the above and do it recursively to handle nested dependencies, etc, and you're done.
2010-12-11 04:40:05 -08:00
me = state.File(name=target)
Whoops, redo-oob was slightly wrong when used with -j. We called 'redo' instead of 'redo-ifchange' on our indeterminate objects. Since other instances of redo-oob might be running at the same time, this could cause the same object to get rebuilt more than once unnecessarily. The unit tests caught this, I just didn't notice earlier.
2010-12-11 05:50:29 -08:00
os.environ['REDO_NO_OOB'] = '1'
argv = ['redo-ifchange'] + deps
The second half of redo-stamp: out-of-order building. If a depends on b depends on c, and c is dirty but b uses redo-stamp checksums, then 'redo-ifchange a' is indeterminate: we won't know if we need to run a.do unless we first build b, but the script that *normally* runs 'redo-ifchange b' is a.do, and we don't want to run that yet, because we don't know for sure if b is dirty, and we shouldn't build a unless one of its dependencies is dirty. Eek! Luckily, there's a safe solution. If we *know* a is dirty - eg. because a.do or one of its children has definitely changed - then we can just run a.do immediately and there's no problem, even if b is indeterminate, because we were going to run a.do anyhow. If a's dependencies are *not* definitely dirty, and all we have is indeterminate ones like b, then that means a's build process *hasn't changed*, which means its tree of dependencies still includes b, which means we can deduce that if we *did* run a.do, it would end up running b.do. Since we know that anyhow, we can safely just run b.do, which will either b.set_checked() or b.set_changed(). Once that's done, we can re-parse a's dependencies and this time conclusively tell if it needs to be redone or not. Even if it does, b is already up-to-date, so the 'redo-ifchange b' line in a.do will be fast. ...now take all the above and do it recursively to handle nested dependencies, etc, and you're done.
2010-12-11 04:40:05 -08:00
rv = os.spawnvp(os.P_WAIT, argv[0], argv)
if rv:
sys.exit(rv)
Whoops, redo-oob was slightly wrong when used with -j. We called 'redo' instead of 'redo-ifchange' on our indeterminate objects. Since other instances of redo-oob might be running at the same time, this could cause the same object to get rebuilt more than once unnecessarily. The unit tests caught this, I just didn't notice earlier.
2010-12-11 05:50:29 -08:00
# we know our caller already owns the lock on target, so we don't have to
# acquire another one.
The second half of redo-stamp: out-of-order building. If a depends on b depends on c, and c is dirty but b uses redo-stamp checksums, then 'redo-ifchange a' is indeterminate: we won't know if we need to run a.do unless we first build b, but the script that *normally* runs 'redo-ifchange b' is a.do, and we don't want to run that yet, because we don't know for sure if b is dirty, and we shouldn't build a unless one of its dependencies is dirty. Eek! Luckily, there's a safe solution. If we *know* a is dirty - eg. because a.do or one of its children has definitely changed - then we can just run a.do immediately and there's no problem, even if b is indeterminate, because we were going to run a.do anyhow. If a's dependencies are *not* definitely dirty, and all we have is indeterminate ones like b, then that means a's build process *hasn't changed*, which means its tree of dependencies still includes b, which means we can deduce that if we *did* run a.do, it would end up running b.do. Since we know that anyhow, we can safely just run b.do, which will either b.set_checked() or b.set_changed(). Once that's done, we can re-parse a's dependencies and this time conclusively tell if it needs to be redone or not. Even if it does, b is already up-to-date, so the 'redo-ifchange b' line in a.do will be fast. ...now take all the above and do it recursively to handle nested dependencies, etc, and you're done.
2010-12-11 04:40:05 -08:00
os.environ['REDO_UNLOCKED'] = '1'
argv = ['redo-ifchange', target]
rv = os.spawnvp(os.P_WAIT, argv[0], argv)
if rv:
sys.exit(rv)
Reference in a new issue Copy permalink