Details
-
Type:
Story
-
Status: Done
-
Resolution: Done
-
Fix Version/s: None
-
Component/s: base
-
Labels:None
-
Story Points:0.5
-
Epic Link:
-
Sprint:DRP S17-3
-
Team:Data Release Production
Description
When a command-line task runs, it writes the versions of packages currently set up to the repository. We're then unable to run the same task again without matching versions. This helps with reproducibility.
However, when I repeatedly try to run the same task with exactly the same task with exactly the same packages set up, this version checking fails intermittently. The task exits, complaining:
Traceback (most recent call last):
|
File "/Users/jds/Projects/Astronomy/LSST/src/pipe_drivers/bin/constructBias.py", line 4, in <module>
|
BiasTask.parseAndSubmit()
|
File "/Users/jds/Projects/Astronomy/LSST/stack/DarwinX86/ctrl_pool/12.1-7-gb57f33e/python/lsst/ctrl/pool/parallel.py", line 422, in parseAndSubmit
|
if not cls.RunnerClass(cls, batchArgs.parent).precall(batchArgs.parent): # Write config, schema
|
File "/Users/jds/Projects/Astronomy/LSST/stack/DarwinX86/pipe_base/12.1-5-g06c326c+6/python/lsst/pipe/base/cmdLineTask.py", line 303, in precall
|
self._precallImpl(task, parsedCmd)
|
File "/Users/jds/Projects/Astronomy/LSST/stack/DarwinX86/pipe_base/12.1-5-g06c326c+6/python/lsst/pipe/base/cmdLineTask.py", line 285, in _precallImpl
|
task.writePackageVersions(parsedCmd.butler, clobber=parsedCmd.clobberVersions)
|
File "/Users/jds/Projects/Astronomy/LSST/stack/DarwinX86/pipe_base/12.1-5-g06c326c+6/python/lsst/pipe/base/cmdLineTask.py", line 618, in writePackageVersions
|
"); consider using --clobber-versions or --no-versions")
|
lsst.pipe.base.task.TaskError: Version mismatch (meas_algorithms: 12.1-17-g13cfda1+6 with boost=1.60.lsst1+1 eigen=3.2.5.lsst2 vs 12.1-17-g13cfda1+6 with eigen=3.2.5.lsst2 boost=1.60.lsst1+1; coadd_utils: 12.1-1-g5961e7a+70 with boost=1.60.lsst1+1 eigen=3.2.5.lsst2 vs 12.1-1-g5961e7a+70 with eigen=3.2.5.lsst2 boost=1.60.lsst1+1; afw: 12.1-31-gb5bd9ab+1 with boost=1.60.lsst1+1 eigen=3.2.5.lsst2 vs 12.1-31-gb5bd9ab+1 with eigen=3.2.5.lsst2 boost=1.60.lsst1+1); consider using --clobber-versions or --no-versions
|
Note that the versions are the same, but are being reported in a different order (12.1-17-g13cfda1+6 with boost=1.60.lsst1+1 eigen=3.2.5.lsst2 as compared to 12.1-17-g13cfda1+6 with eigen=3.2.5.lsst2 boost=1.60.lsst1+1).
Please fix this so that version checking works reliably.
I'm intending to go ahead and fix this myself (it's easy enough when you figure out the trick), but I'm copying Fritz Mueller since I guess it's fundamentally a task framework issue.