forks/apenwarr-redo
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
apenwarr-redo/redo-ifchange.py

129 lines
4.2 KiB
Python
Raw Normal View History

Move 'redo --ifchange' into 'redo-ifchange' to match djb's style. It does simplify the logic of both redo.py and redo-ifchange.py, I suppose.
2010-11-13 00:45:49 -08:00
#!/usr/bin/python
The mtime of a directory is kind of useless, so don't use it.
2010-12-05 03:58:20 -08:00
import sys, os, errno, stat
redo-ifchange can now be run even if there's no parent redo.
2010-12-11 19:08:53 -08:00
if not sys.argv[1:]:
sys.exit(0) # nothing to do, so we can't possibly do it wrong
import vars_init
vars_init.init(sys.argv[1:])
redo-ifchange.py: don't forget to release jwack tokens on exit. This only really matters if it exited abnormally... but it still matters.
2010-11-21 21:15:24 -08:00
import vars, state, builder, jwack
The only thing in helpers.py that needed vars.py was the log stuff. So put it in its own file. Now it's safer to import and use helpers even if you can't safely touch vars.
2010-12-11 18:32:40 -08:00
from helpers import unlink
from log import debug, debug2, err
Move 'redo --ifchange' into 'redo-ifchange' to match djb's style. It does simplify the logic of both redo.py and redo-ifchange.py, I suppose.
2010-11-13 00:45:49 -08:00
The second half of redo-stamp: out-of-order building. If a depends on b depends on c, and c is dirty but b uses redo-stamp checksums, then 'redo-ifchange a' is indeterminate: we won't know if we need to run a.do unless we first build b, but the script that *normally* runs 'redo-ifchange b' is a.do, and we don't want to run that yet, because we don't know for sure if b is dirty, and we shouldn't build a unless one of its dependencies is dirty. Eek! Luckily, there's a safe solution. If we *know* a is dirty - eg. because a.do or one of its children has definitely changed - then we can just run a.do immediately and there's no problem, even if b is indeterminate, because we were going to run a.do anyhow. If a's dependencies are *not* definitely dirty, and all we have is indeterminate ones like b, then that means a's build process *hasn't changed*, which means its tree of dependencies still includes b, which means we can deduce that if we *did* run a.do, it would end up running b.do. Since we know that anyhow, we can safely just run b.do, which will either b.set_checked() or b.set_changed(). Once that's done, we can re-parse a's dependencies and this time conclusively tell if it needs to be redone or not. Even if it does, b is already up-to-date, so the 'redo-ifchange b' line in a.do will be fast. ...now take all the above and do it recursively to handle nested dependencies, etc, and you're done.
2010-12-11 04:40:05 -08:00
CLEAN = 0
DIRTY = 1
Switch state.py to use sqlite3 instead of filesystem-based stamps. It passes all tests when run serialized, but still gives weird errors (OperationalError: database is locked) when run with -j5. sqlite3 shouldn't be barfing just because the database is locked, since the default timeout is 5 seconds, and it's dying *way* faster than that.
2010-12-07 02:17:22 -08:00
def dirty_deps(f, depth, max_changed):
redo-ifchange: fix relative pathnames printed in debug messages.
2010-12-11 02:13:46 -08:00
if vars.DEBUG >= 1:
redo.py: report when you're trying to rebuild a static file. In redo-ifchange, this might be a good idea, since you might just want to set a dependency on it, so we won't say anything from inside builder.py. But if you're calling redo.py, that means you expect it to be rebuilt, since there's no other reason to try. So print a warning. (This is what make does, more or less.)
2010-12-11 21:19:15 -08:00
debug('%s?%s\n' % (depth, f.nicename()))
Switch state.py to use sqlite3 instead of filesystem-based stamps. It passes all tests when run serialized, but still gives weird errors (OperationalError: database is locked) when run with -j5. sqlite3 shouldn't be barfing just because the database is locked, since the default timeout is 5 seconds, and it's dying *way* faster than that.
2010-12-07 02:17:22 -08:00
If a user manually changes a generated file, don't ever overwrite it. That way the user can modify an auto-generated 'compile' script, for example, and it'll stay modified. If they delete the file, we can then generate it for them again. Also, we have to warn whenever we're doing this, or people might think it's a bug.
2010-12-10 22:42:33 -08:00
if f.failed_runid:
debug('%s-- DIRTY (failed last time)\n' % depth)
The second half of redo-stamp: out-of-order building. If a depends on b depends on c, and c is dirty but b uses redo-stamp checksums, then 'redo-ifchange a' is indeterminate: we won't know if we need to run a.do unless we first build b, but the script that *normally* runs 'redo-ifchange b' is a.do, and we don't want to run that yet, because we don't know for sure if b is dirty, and we shouldn't build a unless one of its dependencies is dirty. Eek! Luckily, there's a safe solution. If we *know* a is dirty - eg. because a.do or one of its children has definitely changed - then we can just run a.do immediately and there's no problem, even if b is indeterminate, because we were going to run a.do anyhow. If a's dependencies are *not* definitely dirty, and all we have is indeterminate ones like b, then that means a's build process *hasn't changed*, which means its tree of dependencies still includes b, which means we can deduce that if we *did* run a.do, it would end up running b.do. Since we know that anyhow, we can safely just run b.do, which will either b.set_checked() or b.set_changed(). Once that's done, we can re-parse a's dependencies and this time conclusively tell if it needs to be redone or not. Even if it does, b is already up-to-date, so the 'redo-ifchange b' line in a.do will be fast. ...now take all the above and do it recursively to handle nested dependencies, etc, and you're done.
2010-12-11 04:40:05 -08:00
return DIRTY
Switch state.py to use sqlite3 instead of filesystem-based stamps. It passes all tests when run serialized, but still gives weird errors (OperationalError: database is locked) when run with -j5. sqlite3 shouldn't be barfing just because the database is locked, since the default timeout is 5 seconds, and it's dying *way* faster than that.
2010-12-07 02:17:22 -08:00
if f.changed_runid == None:
debug('%s-- DIRTY (never built)\n' % depth)
The second half of redo-stamp: out-of-order building. If a depends on b depends on c, and c is dirty but b uses redo-stamp checksums, then 'redo-ifchange a' is indeterminate: we won't know if we need to run a.do unless we first build b, but the script that *normally* runs 'redo-ifchange b' is a.do, and we don't want to run that yet, because we don't know for sure if b is dirty, and we shouldn't build a unless one of its dependencies is dirty. Eek! Luckily, there's a safe solution. If we *know* a is dirty - eg. because a.do or one of its children has definitely changed - then we can just run a.do immediately and there's no problem, even if b is indeterminate, because we were going to run a.do anyhow. If a's dependencies are *not* definitely dirty, and all we have is indeterminate ones like b, then that means a's build process *hasn't changed*, which means its tree of dependencies still includes b, which means we can deduce that if we *did* run a.do, it would end up running b.do. Since we know that anyhow, we can safely just run b.do, which will either b.set_checked() or b.set_changed(). Once that's done, we can re-parse a's dependencies and this time conclusively tell if it needs to be redone or not. Even if it does, b is already up-to-date, so the 'redo-ifchange b' line in a.do will be fast. ...now take all the above and do it recursively to handle nested dependencies, etc, and you're done.
2010-12-11 04:40:05 -08:00
return DIRTY
Switch state.py to use sqlite3 instead of filesystem-based stamps. It passes all tests when run serialized, but still gives weird errors (OperationalError: database is locked) when run with -j5. sqlite3 shouldn't be barfing just because the database is locked, since the default timeout is 5 seconds, and it's dying *way* faster than that.
2010-12-07 02:17:22 -08:00
if f.changed_runid > max_changed:
Oops, earlier state.mark() stuff was a little too radical. If someone else built and marked one of our dependencies, then that dependency would show up as *clean* in a later redo-ifchange, so other dependents of that file wouldn't be rebuilt. We actually have to track two session-specific variables: whether the file has been checked, and whether it was rebuilt. (Or alternatively, whether it was dirty when we checked it the first time. But we store the former.)
2010-11-21 04:14:52 -08:00
debug('%s-- DIRTY (built)\n' % depth)
The second half of redo-stamp: out-of-order building. If a depends on b depends on c, and c is dirty but b uses redo-stamp checksums, then 'redo-ifchange a' is indeterminate: we won't know if we need to run a.do unless we first build b, but the script that *normally* runs 'redo-ifchange b' is a.do, and we don't want to run that yet, because we don't know for sure if b is dirty, and we shouldn't build a unless one of its dependencies is dirty. Eek! Luckily, there's a safe solution. If we *know* a is dirty - eg. because a.do or one of its children has definitely changed - then we can just run a.do immediately and there's no problem, even if b is indeterminate, because we were going to run a.do anyhow. If a's dependencies are *not* definitely dirty, and all we have is indeterminate ones like b, then that means a's build process *hasn't changed*, which means its tree of dependencies still includes b, which means we can deduce that if we *did* run a.do, it would end up running b.do. Since we know that anyhow, we can safely just run b.do, which will either b.set_checked() or b.set_changed(). Once that's done, we can re-parse a's dependencies and this time conclusively tell if it needs to be redone or not. Even if it does, b is already up-to-date, so the 'redo-ifchange b' line in a.do will be fast. ...now take all the above and do it recursively to handle nested dependencies, etc, and you're done.
2010-12-11 04:40:05 -08:00
return DIRTY # has been built more recently than parent
Switch state.py to use sqlite3 instead of filesystem-based stamps. It passes all tests when run serialized, but still gives weird errors (OperationalError: database is locked) when run with -j5. sqlite3 shouldn't be barfing just because the database is locked, since the default timeout is 5 seconds, and it's dying *way* faster than that.
2010-12-07 02:17:22 -08:00
if f.is_checked():
The second half of redo-stamp: out-of-order building. If a depends on b depends on c, and c is dirty but b uses redo-stamp checksums, then 'redo-ifchange a' is indeterminate: we won't know if we need to run a.do unless we first build b, but the script that *normally* runs 'redo-ifchange b' is a.do, and we don't want to run that yet, because we don't know for sure if b is dirty, and we shouldn't build a unless one of its dependencies is dirty. Eek! Luckily, there's a safe solution. If we *know* a is dirty - eg. because a.do or one of its children has definitely changed - then we can just run a.do immediately and there's no problem, even if b is indeterminate, because we were going to run a.do anyhow. If a's dependencies are *not* definitely dirty, and all we have is indeterminate ones like b, then that means a's build process *hasn't changed*, which means its tree of dependencies still includes b, which means we can deduce that if we *did* run a.do, it would end up running b.do. Since we know that anyhow, we can safely just run b.do, which will either b.set_checked() or b.set_changed(). Once that's done, we can re-parse a's dependencies and this time conclusively tell if it needs to be redone or not. Even if it does, b is already up-to-date, so the 'redo-ifchange b' line in a.do will be fast. ...now take all the above and do it recursively to handle nested dependencies, etc, and you're done.
2010-12-11 04:40:05 -08:00
if vars.DEBUG >= 1:
debug('%s-- CLEAN (checked)\n' % depth)
return CLEAN # has already been checked during this session
Switch state.py to use sqlite3 instead of filesystem-based stamps. It passes all tests when run serialized, but still gives weird errors (OperationalError: database is locked) when run with -j5. sqlite3 shouldn't be barfing just because the database is locked, since the default timeout is 5 seconds, and it's dying *way* faster than that.
2010-12-07 02:17:22 -08:00
if not f.stamp:
Extract .redo dir state management stuff into its own file. In preparation for changing the on-disk format eventually, as well as making the main code more readable.
2010-11-19 03:03:05 -08:00
debug('%s-- DIRTY (no stamp)\n' % depth)
The second half of redo-stamp: out-of-order building. If a depends on b depends on c, and c is dirty but b uses redo-stamp checksums, then 'redo-ifchange a' is indeterminate: we won't know if we need to run a.do unless we first build b, but the script that *normally* runs 'redo-ifchange b' is a.do, and we don't want to run that yet, because we don't know for sure if b is dirty, and we shouldn't build a unless one of its dependencies is dirty. Eek! Luckily, there's a safe solution. If we *know* a is dirty - eg. because a.do or one of its children has definitely changed - then we can just run a.do immediately and there's no problem, even if b is indeterminate, because we were going to run a.do anyhow. If a's dependencies are *not* definitely dirty, and all we have is indeterminate ones like b, then that means a's build process *hasn't changed*, which means its tree of dependencies still includes b, which means we can deduce that if we *did* run a.do, it would end up running b.do. Since we know that anyhow, we can safely just run b.do, which will either b.set_checked() or b.set_changed(). Once that's done, we can re-parse a's dependencies and this time conclusively tell if it needs to be redone or not. Even if it does, b is already up-to-date, so the 'redo-ifchange b' line in a.do will be fast. ...now take all the above and do it recursively to handle nested dependencies, etc, and you're done.
2010-12-11 04:40:05 -08:00
return DIRTY
Add a redo-always command: it adds an "always dirty" dependency to your target. This is mostly useless except when combined with redo-stamp... I think.
2010-12-11 07:02:45 -08:00
If a checksummed file is deleted, we should still use redo-oob. We were giving up and rebuilding the toplevel object, which did eventually rebuild our checksummed file, but then the file turned out to be identical to what it was before, so that nobody *else* who depended on it ended up getting rebuilt. So the results were indeterminate. Now we treat it as if its dirtiness is unknown, so we build it using redo-oob before building any of its dependencies.
2010-12-11 05:43:02 -08:00
newstamp = f.read_stamp()
if f.stamp != newstamp:
if newstamp == state.STAMP_MISSING:
debug('%s-- DIRTY (missing)\n' % depth)
else:
debug('%s-- DIRTY (mtime)\n' % depth)
if f.csum:
return [f]
else:
return DIRTY
The second half of redo-stamp: out-of-order building. If a depends on b depends on c, and c is dirty but b uses redo-stamp checksums, then 'redo-ifchange a' is indeterminate: we won't know if we need to run a.do unless we first build b, but the script that *normally* runs 'redo-ifchange b' is a.do, and we don't want to run that yet, because we don't know for sure if b is dirty, and we shouldn't build a unless one of its dependencies is dirty. Eek! Luckily, there's a safe solution. If we *know* a is dirty - eg. because a.do or one of its children has definitely changed - then we can just run a.do immediately and there's no problem, even if b is indeterminate, because we were going to run a.do anyhow. If a's dependencies are *not* definitely dirty, and all we have is indeterminate ones like b, then that means a's build process *hasn't changed*, which means its tree of dependencies still includes b, which means we can deduce that if we *did* run a.do, it would end up running b.do. Since we know that anyhow, we can safely just run b.do, which will either b.set_checked() or b.set_changed(). Once that's done, we can re-parse a's dependencies and this time conclusively tell if it needs to be redone or not. Even if it does, b is already up-to-date, so the 'redo-ifchange b' line in a.do will be fast. ...now take all the above and do it recursively to handle nested dependencies, etc, and you're done.
2010-12-11 04:40:05 -08:00
must_build = []
Calculate dependencies with fewer sqlite queries.
2010-12-09 02:13:36 -08:00
for mode,f2 in f.deps():
The second half of redo-stamp: out-of-order building. If a depends on b depends on c, and c is dirty but b uses redo-stamp checksums, then 'redo-ifchange a' is indeterminate: we won't know if we need to run a.do unless we first build b, but the script that *normally* runs 'redo-ifchange b' is a.do, and we don't want to run that yet, because we don't know for sure if b is dirty, and we shouldn't build a unless one of its dependencies is dirty. Eek! Luckily, there's a safe solution. If we *know* a is dirty - eg. because a.do or one of its children has definitely changed - then we can just run a.do immediately and there's no problem, even if b is indeterminate, because we were going to run a.do anyhow. If a's dependencies are *not* definitely dirty, and all we have is indeterminate ones like b, then that means a's build process *hasn't changed*, which means its tree of dependencies still includes b, which means we can deduce that if we *did* run a.do, it would end up running b.do. Since we know that anyhow, we can safely just run b.do, which will either b.set_checked() or b.set_changed(). Once that's done, we can re-parse a's dependencies and this time conclusively tell if it needs to be redone or not. Even if it does, b is already up-to-date, so the 'redo-ifchange b' line in a.do will be fast. ...now take all the above and do it recursively to handle nested dependencies, etc, and you're done.
2010-12-11 04:40:05 -08:00
dirty = CLEAN
Move 'redo --ifchange' into 'redo-ifchange' to match djb's style. It does simplify the logic of both redo.py and redo-ifchange.py, I suppose.
2010-11-13 00:45:49 -08:00
if mode == 'c':
Calculate dependencies with fewer sqlite queries.
2010-12-09 02:13:36 -08:00
if os.path.exists(os.path.join(vars.BASE, f2.name)):
Move 'redo --ifchange' into 'redo-ifchange' to match djb's style. It does simplify the logic of both redo.py and redo-ifchange.py, I suppose.
2010-11-13 00:45:49 -08:00
debug('%s-- DIRTY (created)\n' % depth)
The second half of redo-stamp: out-of-order building. If a depends on b depends on c, and c is dirty but b uses redo-stamp checksums, then 'redo-ifchange a' is indeterminate: we won't know if we need to run a.do unless we first build b, but the script that *normally* runs 'redo-ifchange b' is a.do, and we don't want to run that yet, because we don't know for sure if b is dirty, and we shouldn't build a unless one of its dependencies is dirty. Eek! Luckily, there's a safe solution. If we *know* a is dirty - eg. because a.do or one of its children has definitely changed - then we can just run a.do immediately and there's no problem, even if b is indeterminate, because we were going to run a.do anyhow. If a's dependencies are *not* definitely dirty, and all we have is indeterminate ones like b, then that means a's build process *hasn't changed*, which means its tree of dependencies still includes b, which means we can deduce that if we *did* run a.do, it would end up running b.do. Since we know that anyhow, we can safely just run b.do, which will either b.set_checked() or b.set_changed(). Once that's done, we can re-parse a's dependencies and this time conclusively tell if it needs to be redone or not. Even if it does, b is already up-to-date, so the 'redo-ifchange b' line in a.do will be fast. ...now take all the above and do it recursively to handle nested dependencies, etc, and you're done.
2010-12-11 04:40:05 -08:00
dirty = DIRTY
Move 'redo --ifchange' into 'redo-ifchange' to match djb's style. It does simplify the logic of both redo.py and redo-ifchange.py, I suppose.
2010-11-13 00:45:49 -08:00
elif mode == 'm':
The second half of redo-stamp: out-of-order building. If a depends on b depends on c, and c is dirty but b uses redo-stamp checksums, then 'redo-ifchange a' is indeterminate: we won't know if we need to run a.do unless we first build b, but the script that *normally* runs 'redo-ifchange b' is a.do, and we don't want to run that yet, because we don't know for sure if b is dirty, and we shouldn't build a unless one of its dependencies is dirty. Eek! Luckily, there's a safe solution. If we *know* a is dirty - eg. because a.do or one of its children has definitely changed - then we can just run a.do immediately and there's no problem, even if b is indeterminate, because we were going to run a.do anyhow. If a's dependencies are *not* definitely dirty, and all we have is indeterminate ones like b, then that means a's build process *hasn't changed*, which means its tree of dependencies still includes b, which means we can deduce that if we *did* run a.do, it would end up running b.do. Since we know that anyhow, we can safely just run b.do, which will either b.set_checked() or b.set_changed(). Once that's done, we can re-parse a's dependencies and this time conclusively tell if it needs to be redone or not. Even if it does, b is already up-to-date, so the 'redo-ifchange b' line in a.do will be fast. ...now take all the above and do it recursively to handle nested dependencies, etc, and you're done.
2010-12-11 04:40:05 -08:00
sub = dirty_deps(f2, depth = depth + ' ',
max_changed = max(f.changed_runid,
f.checked_runid))
if sub:
Oops, earlier state.mark() stuff was a little too radical. If someone else built and marked one of our dependencies, then that dependency would show up as *clean* in a later redo-ifchange, so other dependents of that file wouldn't be rebuilt. We actually have to track two session-specific variables: whether the file has been checked, and whether it was rebuilt. (Or alternatively, whether it was dirty when we checked it the first time. But we store the former.)
2010-11-21 04:14:52 -08:00
debug('%s-- DIRTY (sub)\n' % depth)
The second half of redo-stamp: out-of-order building. If a depends on b depends on c, and c is dirty but b uses redo-stamp checksums, then 'redo-ifchange a' is indeterminate: we won't know if we need to run a.do unless we first build b, but the script that *normally* runs 'redo-ifchange b' is a.do, and we don't want to run that yet, because we don't know for sure if b is dirty, and we shouldn't build a unless one of its dependencies is dirty. Eek! Luckily, there's a safe solution. If we *know* a is dirty - eg. because a.do or one of its children has definitely changed - then we can just run a.do immediately and there's no problem, even if b is indeterminate, because we were going to run a.do anyhow. If a's dependencies are *not* definitely dirty, and all we have is indeterminate ones like b, then that means a's build process *hasn't changed*, which means its tree of dependencies still includes b, which means we can deduce that if we *did* run a.do, it would end up running b.do. Since we know that anyhow, we can safely just run b.do, which will either b.set_checked() or b.set_changed(). Once that's done, we can re-parse a's dependencies and this time conclusively tell if it needs to be redone or not. Even if it does, b is already up-to-date, so the 'redo-ifchange b' line in a.do will be fast. ...now take all the above and do it recursively to handle nested dependencies, etc, and you're done.
2010-12-11 04:40:05 -08:00
dirty = sub
else:
assert(mode in ('c','m'))
if not f.csum:
# f is a "normal" target: dirty f2 means f is instantly dirty
if dirty:
# if dirty==DIRTY, this means f is definitely dirty.
# if dirty==[...], it's a list of the uncertain children.
return dirty
else:
# f is "checksummable": dirty f2 means f needs to redo,
# but f might turn out to be clean after that (ie. our parent
# might not be dirty).
if dirty == DIRTY:
# f2 is definitely dirty, so f definitely needs to
# redo. However, after that, f might turn out to be
# unchanged.
return [f]
elif isinstance(dirty,list):
# our child f2 might be dirty, but it's not sure yet. It's
# given us a list of targets we have to redo in order to
# be sure.
must_build += dirty
if must_build:
# f is *maybe* dirty because at least one of its children is maybe
# dirty. must_build has accumulated a list of "topmost" uncertain
# objects in the tree. If we build all those, we can then
# redo-ifchange f and it won't have any uncertainty next time.
return must_build
# if we get here, it's because the target is clean
If a user manually changes a generated file, don't ever overwrite it. That way the user can modify an auto-generated 'compile' script, for example, and it'll stay modified. If they delete the file, we can then generate it for them again. Also, we have to warn whenever we're doing this, or people might think it's a bug.
2010-12-10 22:42:33 -08:00
if f.is_override:
builder.warn_override(f.name)
Switch state.py to use sqlite3 instead of filesystem-based stamps. It passes all tests when run serialized, but still gives weird errors (OperationalError: database is locked) when run with -j5. sqlite3 shouldn't be barfing just because the database is locked, since the default timeout is 5 seconds, and it's dying *way* faster than that.
2010-12-07 02:17:22 -08:00
f.set_checked()
f.save()
The second half of redo-stamp: out-of-order building. If a depends on b depends on c, and c is dirty but b uses redo-stamp checksums, then 'redo-ifchange a' is indeterminate: we won't know if we need to run a.do unless we first build b, but the script that *normally* runs 'redo-ifchange b' is a.do, and we don't want to run that yet, because we don't know for sure if b is dirty, and we shouldn't build a unless one of its dependencies is dirty. Eek! Luckily, there's a safe solution. If we *know* a is dirty - eg. because a.do or one of its children has definitely changed - then we can just run a.do immediately and there's no problem, even if b is indeterminate, because we were going to run a.do anyhow. If a's dependencies are *not* definitely dirty, and all we have is indeterminate ones like b, then that means a's build process *hasn't changed*, which means its tree of dependencies still includes b, which means we can deduce that if we *did* run a.do, it would end up running b.do. Since we know that anyhow, we can safely just run b.do, which will either b.set_checked() or b.set_changed(). Once that's done, we can re-parse a's dependencies and this time conclusively tell if it needs to be redone or not. Even if it does, b is already up-to-date, so the 'redo-ifchange b' line in a.do will be fast. ...now take all the above and do it recursively to handle nested dependencies, etc, and you're done.
2010-12-11 04:40:05 -08:00
return CLEAN
Move 'redo --ifchange' into 'redo-ifchange' to match djb's style. It does simplify the logic of both redo.py and redo-ifchange.py, I suppose.
2010-11-13 00:45:49 -08:00
Oops, earlier state.mark() stuff was a little too radical. If someone else built and marked one of our dependencies, then that dependency would show up as *clean* in a later redo-ifchange, so other dependents of that file wouldn't be rebuilt. We actually have to track two session-specific variables: whether the file has been checked, and whether it was rebuilt. (Or alternatively, whether it was dirty when we checked it the first time. But we store the former.)
2010-11-21 04:14:52 -08:00
def should_build(t):
Switch state.py to use sqlite3 instead of filesystem-based stamps. It passes all tests when run serialized, but still gives weird errors (OperationalError: database is locked) when run with -j5. sqlite3 shouldn't be barfing just because the database is locked, since the default timeout is 5 seconds, and it's dying *way* faster than that.
2010-12-07 02:17:22 -08:00
f = state.File(name=t)
Don't wipe the timestamp when a target fails to redo. It's really a separate condition. And since we're not removing the target *file* in case of error - we update it atomically, and keeping it is better than losing it - there's no reason to wipe the timestamp in that case either. However, we do need to know that the build failed, so that anybody else (especially in a parallel build) who looks at that target knows that it died. So add a separate flag just for that.
2010-12-10 20:53:31 -08:00
if f.is_failed():
raise builder.ImmediateReturn(32)
Fix a deadlock with redo-oob. If a checksummed target A used to exist but is now missing, and we tried to redo-ifchange that exact file, we would unnecessarily run 'redo-oob A A'; that is, we have to build A in order to determine if A needs to be built. The sub-targets of redo-oob aren't run with REDO_UNLOCKED, so this would deadlock instantly. Add an assertion to redo-oob to ensure we never try to redo-ifchange the primary target (thus converting the deadlock into an exception). And skip doing redo-oob when the target is already the same as the thing we have to check.
2010-12-11 06:16:32 -08:00
dirty = dirty_deps(f, depth = '', max_changed = vars.RUNID)
return dirty==[f] and DIRTY or dirty
Oops, earlier state.mark() stuff was a little too radical. If someone else built and marked one of our dependencies, then that dependency would show up as *clean* in a later redo-ifchange, so other dependents of that file wouldn't be rebuilt. We actually have to track two session-specific variables: whether the file has been checked, and whether it was rebuilt. (Or alternatively, whether it was dirty when we checked it the first time. But we store the former.)
2010-11-21 04:14:52 -08:00
redo-ifchange: call builder directly, rather than forking redo. Now we no longer rebuild the same stuff over and over!
2010-11-19 07:32:16 -08:00
rv = 202
Handle KeyboardInterrupt with less annoying spew to stderr.
2010-11-13 02:09:42 -08:00
try:
redo-ifchange can now be run even if there's no parent redo.
2010-12-11 19:08:53 -08:00
if vars.TARGET and not vars.UNLOCKED:
me = os.path.join(vars.STARTDIR,
os.path.join(vars.PWD, vars.TARGET))
f = state.File(name=me)
debug2('TARGET: %r %r %r\n' % (vars.STARTDIR, vars.PWD, vars.TARGET))
else:
f = me = None
debug2('redo-ifchange: not adding depends.\n')
redo-ifchange.py: don't forget to release jwack tokens on exit. This only really matters if it exited abnormally... but it still matters.
2010-11-21 21:15:24 -08:00
try:
builder.py: now the only exported function is main(). We can also avoid forking altogether if should_build() returns false. This doesn't seem to result in any noticeable speedup, but it's cleaner at least.
2010-11-21 23:33:11 -08:00
targets = sys.argv[1:]
redo-ifchange can now be run even if there's no parent redo.
2010-12-11 19:08:53 -08:00
if f:
The second half of redo-stamp: out-of-order building. If a depends on b depends on c, and c is dirty but b uses redo-stamp checksums, then 'redo-ifchange a' is indeterminate: we won't know if we need to run a.do unless we first build b, but the script that *normally* runs 'redo-ifchange b' is a.do, and we don't want to run that yet, because we don't know for sure if b is dirty, and we shouldn't build a unless one of its dependencies is dirty. Eek! Luckily, there's a safe solution. If we *know* a is dirty - eg. because a.do or one of its children has definitely changed - then we can just run a.do immediately and there's no problem, even if b is indeterminate, because we were going to run a.do anyhow. If a's dependencies are *not* definitely dirty, and all we have is indeterminate ones like b, then that means a's build process *hasn't changed*, which means its tree of dependencies still includes b, which means we can deduce that if we *did* run a.do, it would end up running b.do. Since we know that anyhow, we can safely just run b.do, which will either b.set_checked() or b.set_changed(). Once that's done, we can re-parse a's dependencies and this time conclusively tell if it needs to be redone or not. Even if it does, b is already up-to-date, so the 'redo-ifchange b' line in a.do will be fast. ...now take all the above and do it recursively to handle nested dependencies, etc, and you're done.
2010-12-11 04:40:05 -08:00
for t in targets:
f.add_dep('m', t)
f.save()
builder.py: now the only exported function is main(). We can also avoid forking altogether if should_build() returns false. This doesn't seem to result in any noticeable speedup, but it's cleaner at least.
2010-11-21 23:33:11 -08:00
rv = builder.main(targets, should_build)
redo-ifchange.py: don't forget to release jwack tokens on exit. This only really matters if it exited abnormally... but it still matters.
2010-11-21 21:15:24 -08:00
finally:
jwack.force_return_tokens()
Handle KeyboardInterrupt with less annoying spew to stderr.
2010-11-13 02:09:42 -08:00
except KeyboardInterrupt:
sys.exit(200)
redo-ifchange can now be run even if there's no parent redo.
2010-12-11 19:08:53 -08:00
state.commit()
redo-ifchange: call builder directly, rather than forking redo. Now we no longer rebuild the same stuff over and over!
2010-11-19 07:32:16 -08:00
sys.exit(rv)
Reference in a new issue Copy permalink