Details
-
Type:
Story
-
Status: Done
-
Resolution: Done
-
Fix Version/s: None
-
Component/s: None
-
Labels:None
-
Story Points:8
-
Epic Link:
-
Sprint:AP F21-6 (November), AP S22-1 (December)
-
Team:Alert Production
-
Urgent?:No
Description
As a followup to DM-30703, reprocess HiTS with multi-tract fakes support, and ensure the diaPipe step actually writes to the APDB.
Attachments
Issue Links
Activity
I did a test run with the DM-31964 branch today on tracts 9812 and 9813. I had to rename (actually "relabel") the tasks originally referred to as visitFakes and matchFakes in the pipeline yaml since the dimensions changed. After the quantum graph took its time building, I got this error:
(lsst-scipipe) [mrawls@lsst-devl03 repo-main-logs]$ more /project/mrawls/repo-main-logs/bps/u/mrawls/DM-31964-DECam-fakes/20211124T201946Z/quantumGraphGeneration.out
|
/software/lsstsw/stack_20210813/stack/miniconda3-py38_4.9.2-0.7.0/Linux64/ctrl_mpexec/22.0.1-29-gedb21c8+82be1abe8f/bin/pipetask qgraph -d "instrument='DECam' AND skymap='hsc_rings_v1' AND
|
band IN ('g') AND exposure>410914 AND exposure<421607 AND tract IN (9812, 9813)" -b /repo/main/butler.yaml -i DECam/defaults/hits2015,DECam/calib,u/mrawls/DM-30703-hits-crosstalk-sources,u/
|
mrawls/DM-30703-HiTS2014-templates -p /project/mrawls/repo-main-logs/ApPipe_DM32245.yaml -q /project/mrawls/repo-main-logs/bps/u/mrawls/DM-31964-DECam-fakes/20211124T201946Z/u_mrawls_DM-319
|
64-DECam-fakes_20211124T201946Z.qgraph
|
|
|
|
|
lsst.daf.butler.cli.utils ERROR: Caught an exception, details are in traceback:
|
Traceback (most recent call last):
|
File "/software/lsstsw/stack_20210813/stack/miniconda3-py38_4.9.2-0.7.0/Linux64/ctrl_mpexec/22.0.1-29-gedb21c8+82be1abe8f/python/lsst/ctrl/mpexec/cli/cmd/commands.py", line 96, in qgraph
|
script.qgraph(pipelineObj=pipeline, **kwargs)
|
File "/software/lsstsw/stack_20210813/stack/miniconda3-py38_4.9.2-0.7.0/Linux64/ctrl_mpexec/22.0.1-29-gedb21c8+82be1abe8f/python/lsst/ctrl/mpexec/cli/script/qgraph.py", line 156, in qgrap
|
h
|
qgraph = f.makeGraph(pipelineObj, args)
|
File "/software/lsstsw/stack_20210813/stack/miniconda3-py38_4.9.2-0.7.0/Linux64/ctrl_mpexec/22.0.1-29-gedb21c8+82be1abe8f/python/lsst/ctrl/mpexec/cmdLineFwk.py", line 571, in makeGraph
|
qgraph = graphBuilder.makeGraph(pipeline, collections, run, args.data_query, metadata=metadata,
|
File "/software/lsstsw/stack_20210813/stack/miniconda3-py38_4.9.2-0.7.0/Linux64/pipe_base/22.0.1-32-g19dc4ec+303e5401d2/python/lsst/pipe/base/graphBuilder.py", line 1043, in makeGraph
|
return scaffolding.makeQuantumGraph(metadata=metadata)
|
File "/software/lsstsw/stack_20210813/stack/miniconda3-py38_4.9.2-0.7.0/Linux64/pipe_base/22.0.1-32-g19dc4ec+303e5401d2/python/lsst/pipe/base/graphBuilder.py", line 929, in makeQuantumGra
|
ph
|
graph = QuantumGraph(graphInput, metadata=metadata, pruneRefs=self.unfoundRefs)
|
File "/software/lsstsw/stack_20210813/stack/miniconda3-py38_4.9.2-0.7.0/Linux64/pipe_base/22.0.1-32-g19dc4ec+303e5401d2/python/lsst/pipe/base/graph/graph.py", line 114, in __init__
|
self._buildGraphs(quanta, metadata=metadata, pruneRefs=pruneRefs)
|
File "/software/lsstsw/stack_20210813/stack/miniconda3-py38_4.9.2-0.7.0/Linux64/pipe_base/22.0.1-32-g19dc4ec+303e5401d2/python/lsst/pipe/base/graph/graph.py", line 184, in _buildGraphs
|
self._datasetRefDict.addProducer(dsRef, value)
|
File "/software/lsstsw/stack_20210813/stack/miniconda3-py38_4.9.2-0.7.0/Linux64/pipe_base/22.0.1-32-g19dc4ec+303e5401d2/python/lsst/pipe/base/graph/_implDetails.py", line 82, in addProduc
|
er
|
raise ValueError(f"Only one node is allowed to produce {key}, "
|
ValueError: Only one node is allowed to produce fakes_src@{instrument: 'DECam', detector: 51, visit: 412504, ...}, sc=SourceCatalog], the current producer is QuantumNode(quantum=Quantum(tas
|
kName=lsst.pipe.tasks.processCcdWithFakes.ProcessCcdWithFakesTask, dataId={instrument: 'DECam', skymap: 'hsc_rings_v1', detector: 51, tract: 9812, visit: 412504, ...}), taskDef=TaskDef(Proc
|
essCcdWithFakesTask, label=processVisitFakes), nodeId=NodeId(number=5084, buildId='1637786834.584632-3801835'))
|
I got all the tracts to process with the pipe_tasks ticket branch of DM-31964! The results are in /repo/main in collection u/mrawls/DM-32245-DECam-fakes-all. The APDB is in postgres lsstdevapdb1, schema mrawls_DM32245.
I created new ApPipeMultiTractFakes.yaml pipelines in ap_pipe, and tested they work for each of DECam, HSC, and LSSTCam-imSim with w_2021_49 and the pipe_tasks DM-31964 ticket branch. (I realize creating fakes in DC2 is close to pointless, but this served as an integration test for the multi-tract image differencing workflow as well as fakes.)
These are the commands I ran for integration testing the new pipelines (all were run from /project/mrawls/repo-main-logs):
nohup pipetask run -b /repo/main -p AP_PIPE_DIR/pipelines/DarkEnergyCamera/ApPipeMultiTractFakes.yaml -i DECam/defaults/hits2015,DECam/calib,u/mrawls/DM-30703-hits-crosstalk-sources,u/mrawls/DM-30703-HiTS2014-templates -o u/mrawls/DM32245-decamtest -c diaPipe:apdb.db_url="sqlite3:////project/mrawls/repo-main-logs/association-decam-test.db" -d "instrument='DECam' AND skymap='hsc_rings_v1' AND band IN ('g') AND detector=51 AND visit=412504" > decam-test.out & |
|
|
nohup pipetask run -b /repo/main -p $AP_PIPE_DIR/pipelines/HyperSuprimeCam/ApPipeMultiTractFakes.yaml -i u/mrawls/DM-31053-HSC-fakes -o u/mrawls/DM32245-hsctest -c diaPipe:apdb.db_url="sqlite:////project/mrawls/repo-main-logs/association-hsc-test.db" -d "instrument='HSC' AND skymap='hsc_rings_v1' AND band IN ('g') AND detector<10 AND visit=11690" > hsc-test.out & |
|
|
nohup pipetask run -b /repo/dc2 -p $AP_PIPE_DIR/pipelines/LsstCamImSim/ApPipeMultiTractFakes.yaml -i u/mrawls/DM-30802-AP -o u/mrawls/DM32245-lssttest -c diaPipe:apdb.db_url="sqlite:////project/mrawls/repo-main-logs/association-lsst-test.db" -d "instrument='LSSTCam-imSim' AND skymap='DC2' AND detector<100 AND exposure=214433" --register-dataset-types > lsst-test.out & |
Finally, I made an analysis notebook for the full run.
This is ready for review, thanks for agreeing to do it Krzysztof Findeisen. There are some pipelines updates in ap_pipe, as well as minor plotting script updates and a new analysis notebook in ap_pipe-notebooks.
You are also welcome to peruse the outputs in the APDB or the output collection (the exact name of each are in the previous comment).
There are four precursor collections, run as separate tracts:
DM-32245-DECam-fakes-9812DM-32245-DECam-fakes-9813DM-32245-DECam-fakes-8604DM-32245-DECam-fakes-8605As reported on
DM-32246, they basically ran fine, except whichever overlapping tract ran second had many (mostly expected) diaPipe failures due to duplicate DiaObjects keys.I will pick up on this ticket once
DM-31964is complete.