Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-32245

Reprocess HiTS AP with fakes and an APDB

    XMLWordPrintable

Details

    • Story
    • Status: Done
    • Resolution: Done
    • None
    • None
    • None
    • 8
    • AP F21-6 (November), AP S22-1 (December)
    • Alert Production
    • No

    Description

      As a followup to DM-30703, reprocess HiTS with multi-tract fakes support, and ensure the diaPipe step actually writes to the APDB.

      Attachments

        Issue Links

          Activity

            There are four precursor collections, run as separate tracts:

            As reported on DM-32246, they basically ran fine, except whichever overlapping tract ran second had many (mostly expected) diaPipe failures due to duplicate DiaObjects keys.

            I will pick up on this ticket once DM-31964 is complete.

            mrawls Meredith Rawls added a comment - There are four precursor collections, run as separate tracts: u/mrawls/ DM-32245 -DECam-fakes-9812 u/mrawls/ DM-32245 -DECam-fakes-9813 u/mrawls/ DM-32245 -DECam-fakes-8604 u/mrawls/ DM-32245 -DECam-fakes-8605 As reported on DM-32246 , they basically ran fine, except whichever overlapping tract ran second had many (mostly expected) diaPipe failures due to duplicate DiaObjects keys. I will pick up on this ticket once DM-31964 is complete.

            I did a test run with the DM-31964 branch today on tracts 9812 and 9813. I had to rename (actually "relabel") the tasks originally referred to as visitFakes and matchFakes in the pipeline yaml since the dimensions changed. After the quantum graph took its time building, I got this error:

            (lsst-scipipe) [mrawls@lsst-devl03 repo-main-logs]$ more /project/mrawls/repo-main-logs/bps/u/mrawls/DM-31964-DECam-fakes/20211124T201946Z/quantumGraphGeneration.out
            /software/lsstsw/stack_20210813/stack/miniconda3-py38_4.9.2-0.7.0/Linux64/ctrl_mpexec/22.0.1-29-gedb21c8+82be1abe8f/bin/pipetask qgraph -d "instrument='DECam' AND skymap='hsc_rings_v1' AND 
            band IN ('g') AND exposure>410914 AND exposure<421607 AND tract IN (9812, 9813)" -b /repo/main/butler.yaml -i DECam/defaults/hits2015,DECam/calib,u/mrawls/DM-30703-hits-crosstalk-sources,u/
            mrawls/DM-30703-HiTS2014-templates -p /project/mrawls/repo-main-logs/ApPipe_DM32245.yaml -q /project/mrawls/repo-main-logs/bps/u/mrawls/DM-31964-DECam-fakes/20211124T201946Z/u_mrawls_DM-319
            64-DECam-fakes_20211124T201946Z.qgraph
             
             
            lsst.daf.butler.cli.utils ERROR: Caught an exception, details are in traceback:
            Traceback (most recent call last):
              File "/software/lsstsw/stack_20210813/stack/miniconda3-py38_4.9.2-0.7.0/Linux64/ctrl_mpexec/22.0.1-29-gedb21c8+82be1abe8f/python/lsst/ctrl/mpexec/cli/cmd/commands.py", line 96, in qgraph
                script.qgraph(pipelineObj=pipeline, **kwargs)
              File "/software/lsstsw/stack_20210813/stack/miniconda3-py38_4.9.2-0.7.0/Linux64/ctrl_mpexec/22.0.1-29-gedb21c8+82be1abe8f/python/lsst/ctrl/mpexec/cli/script/qgraph.py", line 156, in qgrap
            h
                qgraph = f.makeGraph(pipelineObj, args)
              File "/software/lsstsw/stack_20210813/stack/miniconda3-py38_4.9.2-0.7.0/Linux64/ctrl_mpexec/22.0.1-29-gedb21c8+82be1abe8f/python/lsst/ctrl/mpexec/cmdLineFwk.py", line 571, in makeGraph
                qgraph = graphBuilder.makeGraph(pipeline, collections, run, args.data_query, metadata=metadata,
              File "/software/lsstsw/stack_20210813/stack/miniconda3-py38_4.9.2-0.7.0/Linux64/pipe_base/22.0.1-32-g19dc4ec+303e5401d2/python/lsst/pipe/base/graphBuilder.py", line 1043, in makeGraph
                return scaffolding.makeQuantumGraph(metadata=metadata)
              File "/software/lsstsw/stack_20210813/stack/miniconda3-py38_4.9.2-0.7.0/Linux64/pipe_base/22.0.1-32-g19dc4ec+303e5401d2/python/lsst/pipe/base/graphBuilder.py", line 929, in makeQuantumGra
            ph
                graph = QuantumGraph(graphInput, metadata=metadata, pruneRefs=self.unfoundRefs)
              File "/software/lsstsw/stack_20210813/stack/miniconda3-py38_4.9.2-0.7.0/Linux64/pipe_base/22.0.1-32-g19dc4ec+303e5401d2/python/lsst/pipe/base/graph/graph.py", line 114, in __init__
                self._buildGraphs(quanta, metadata=metadata, pruneRefs=pruneRefs)
              File "/software/lsstsw/stack_20210813/stack/miniconda3-py38_4.9.2-0.7.0/Linux64/pipe_base/22.0.1-32-g19dc4ec+303e5401d2/python/lsst/pipe/base/graph/graph.py", line 184, in _buildGraphs
                self._datasetRefDict.addProducer(dsRef, value)
              File "/software/lsstsw/stack_20210813/stack/miniconda3-py38_4.9.2-0.7.0/Linux64/pipe_base/22.0.1-32-g19dc4ec+303e5401d2/python/lsst/pipe/base/graph/_implDetails.py", line 82, in addProduc
            er
                raise ValueError(f"Only one node is allowed to produce {key}, "
            ValueError: Only one node is allowed to produce fakes_src@{instrument: 'DECam', detector: 51, visit: 412504, ...}, sc=SourceCatalog], the current producer is QuantumNode(quantum=Quantum(tas
            kName=lsst.pipe.tasks.processCcdWithFakes.ProcessCcdWithFakesTask, dataId={instrument: 'DECam', skymap: 'hsc_rings_v1', detector: 51, tract: 9812, visit: 412504, ...}), taskDef=TaskDef(Proc
            essCcdWithFakesTask, label=processVisitFakes), nodeId=NodeId(number=5084, buildId='1637786834.584632-3801835'))
            

            mrawls Meredith Rawls added a comment - I did a test run with the DM-31964 branch today on tracts 9812 and 9813. I had to rename (actually "relabel") the tasks originally referred to as visitFakes and matchFakes in the pipeline yaml since the dimensions changed. After the quantum graph took its time building, I got this error: (lsst-scipipe) [mrawls@lsst-devl03 repo-main-logs]$ more /project/mrawls/repo-main-logs/bps/u/mrawls/DM-31964-DECam-fakes/20211124T201946Z/quantumGraphGeneration.out /software/lsstsw/stack_20210813/stack/miniconda3-py38_4.9.2-0.7.0/Linux64/ctrl_mpexec/22.0.1-29-gedb21c8+82be1abe8f/bin/pipetask qgraph -d "instrument='DECam' AND skymap='hsc_rings_v1' AND band IN ('g') AND exposure>410914 AND exposure<421607 AND tract IN (9812, 9813)" -b /repo/main/butler.yaml -i DECam/defaults/hits2015,DECam/calib,u/mrawls/DM-30703-hits-crosstalk-sources,u/ mrawls/DM-30703-HiTS2014-templates -p /project/mrawls/repo-main-logs/ApPipe_DM32245.yaml -q /project/mrawls/repo-main-logs/bps/u/mrawls/DM-31964-DECam-fakes/20211124T201946Z/u_mrawls_DM-319 64-DECam-fakes_20211124T201946Z.qgraph     lsst.daf.butler.cli.utils ERROR: Caught an exception, details are in traceback: Traceback (most recent call last): File "/software/lsstsw/stack_20210813/stack/miniconda3-py38_4.9.2-0.7.0/Linux64/ctrl_mpexec/22.0.1-29-gedb21c8+82be1abe8f/python/lsst/ctrl/mpexec/cli/cmd/commands.py", line 96, in qgraph script.qgraph(pipelineObj=pipeline, **kwargs) File "/software/lsstsw/stack_20210813/stack/miniconda3-py38_4.9.2-0.7.0/Linux64/ctrl_mpexec/22.0.1-29-gedb21c8+82be1abe8f/python/lsst/ctrl/mpexec/cli/script/qgraph.py", line 156, in qgrap h qgraph = f.makeGraph(pipelineObj, args) File "/software/lsstsw/stack_20210813/stack/miniconda3-py38_4.9.2-0.7.0/Linux64/ctrl_mpexec/22.0.1-29-gedb21c8+82be1abe8f/python/lsst/ctrl/mpexec/cmdLineFwk.py", line 571, in makeGraph qgraph = graphBuilder.makeGraph(pipeline, collections, run, args.data_query, metadata=metadata, File "/software/lsstsw/stack_20210813/stack/miniconda3-py38_4.9.2-0.7.0/Linux64/pipe_base/22.0.1-32-g19dc4ec+303e5401d2/python/lsst/pipe/base/graphBuilder.py", line 1043, in makeGraph return scaffolding.makeQuantumGraph(metadata=metadata) File "/software/lsstsw/stack_20210813/stack/miniconda3-py38_4.9.2-0.7.0/Linux64/pipe_base/22.0.1-32-g19dc4ec+303e5401d2/python/lsst/pipe/base/graphBuilder.py", line 929, in makeQuantumGra ph graph = QuantumGraph(graphInput, metadata=metadata, pruneRefs=self.unfoundRefs) File "/software/lsstsw/stack_20210813/stack/miniconda3-py38_4.9.2-0.7.0/Linux64/pipe_base/22.0.1-32-g19dc4ec+303e5401d2/python/lsst/pipe/base/graph/graph.py", line 114, in __init__ self._buildGraphs(quanta, metadata=metadata, pruneRefs=pruneRefs) File "/software/lsstsw/stack_20210813/stack/miniconda3-py38_4.9.2-0.7.0/Linux64/pipe_base/22.0.1-32-g19dc4ec+303e5401d2/python/lsst/pipe/base/graph/graph.py", line 184, in _buildGraphs self._datasetRefDict.addProducer(dsRef, value) File "/software/lsstsw/stack_20210813/stack/miniconda3-py38_4.9.2-0.7.0/Linux64/pipe_base/22.0.1-32-g19dc4ec+303e5401d2/python/lsst/pipe/base/graph/_implDetails.py", line 82, in addProduc er raise ValueError(f"Only one node is allowed to produce {key}, " ValueError: Only one node is allowed to produce fakes_src@{instrument: 'DECam', detector: 51, visit: 412504, ...}, sc=SourceCatalog], the current producer is QuantumNode(quantum=Quantum(tas kName=lsst.pipe.tasks.processCcdWithFakes.ProcessCcdWithFakesTask, dataId={instrument: 'DECam', skymap: 'hsc_rings_v1', detector: 51, tract: 9812, visit: 412504, ...}), taskDef=TaskDef(Proc essCcdWithFakesTask, label=processVisitFakes), nodeId=NodeId(number=5084, buildId='1637786834.584632-3801835'))
            mrawls Meredith Rawls added a comment - - edited

            I got all the tracts to process with the pipe_tasks ticket branch of DM-31964! The results are in /repo/main in collection u/mrawls/DM-32245-DECam-fakes-all. The APDB is in postgres lsstdevapdb1, schema mrawls_DM32245.

            I created new ApPipeMultiTractFakes.yaml pipelines in ap_pipe, and tested they work for each of DECam, HSC, and LSSTCam-imSim with w_2021_49 and the pipe_tasks DM-31964 ticket branch. (I realize creating fakes in DC2 is close to pointless, but this served as an integration test for the multi-tract image differencing workflow as well as fakes.)

            These are the commands I ran for integration testing the new pipelines (all were run from /project/mrawls/repo-main-logs):

            nohup pipetask run -b /repo/main -p AP_PIPE_DIR/pipelines/DarkEnergyCamera/ApPipeMultiTractFakes.yaml -i DECam/defaults/hits2015,DECam/calib,u/mrawls/DM-30703-hits-crosstalk-sources,u/mrawls/DM-30703-HiTS2014-templates -o u/mrawls/DM32245-decamtest -c diaPipe:apdb.db_url="sqlite3:////project/mrawls/repo-main-logs/association-decam-test.db" -d "instrument='DECam' AND skymap='hsc_rings_v1' AND band IN ('g') AND detector=51 AND visit=412504" > decam-test.out &
             
            nohup pipetask run -b /repo/main -p $AP_PIPE_DIR/pipelines/HyperSuprimeCam/ApPipeMultiTractFakes.yaml -i u/mrawls/DM-31053-HSC-fakes -o u/mrawls/DM32245-hsctest -c diaPipe:apdb.db_url="sqlite:////project/mrawls/repo-main-logs/association-hsc-test.db" -d "instrument='HSC' AND skymap='hsc_rings_v1' AND band IN ('g') AND detector<10 AND visit=11690" > hsc-test.out &
             
            nohup pipetask run -b /repo/dc2 -p $AP_PIPE_DIR/pipelines/LsstCamImSim/ApPipeMultiTractFakes.yaml -i u/mrawls/DM-30802-AP -o u/mrawls/DM32245-lssttest -c diaPipe:apdb.db_url="sqlite:////project/mrawls/repo-main-logs/association-lsst-test.db" -d "instrument='LSSTCam-imSim' AND skymap='DC2' AND detector<100 AND exposure=214433" --register-dataset-types > lsst-test.out & 

            Finally, I made an analysis notebook for the full run.

            mrawls Meredith Rawls added a comment - - edited I got all the tracts to process with the pipe_tasks ticket branch of DM-31964 ! The results are in /repo/main in collection u/mrawls/ DM-32245 -DECam-fakes-all. The APDB is in postgres lsstdevapdb1, schema mrawls_DM32245. I created new ApPipeMultiTractFakes.yaml pipelines in ap_pipe, and tested they work for each of DECam, HSC, and LSSTCam-imSim with w_2021_49 and the pipe_tasks DM-31964 ticket branch. (I realize creating fakes in DC2 is close to pointless, but this served as an integration test for the multi-tract image differencing workflow as well as fakes.) These are the commands I ran for integration testing the new pipelines (all were run from /project/mrawls/repo-main-logs): nohup pipetask run -b /repo/main -p AP_PIPE_DIR/pipelines/DarkEnergyCamera/ApPipeMultiTractFakes.yaml -i DECam/defaults/hits2015,DECam/calib,u/mrawls/DM- 30703 -hits-crosstalk-sources,u/mrawls/DM- 30703 -HiTS2014-templates -o u/mrawls/DM32245-decamtest -c diaPipe:apdb.db_url= "sqlite3:////project/mrawls/repo-main-logs/association-decam-test.db" -d "instrument='DECam' AND skymap='hsc_rings_v1' AND band IN ('g') AND detector=51 AND visit=412504" > decam-test.out &   nohup pipetask run -b /repo/main -p $AP_PIPE_DIR/pipelines/HyperSuprimeCam/ApPipeMultiTractFakes.yaml -i u/mrawls/DM- 31053 -HSC-fakes -o u/mrawls/DM32245-hsctest -c diaPipe:apdb.db_url= "sqlite:////project/mrawls/repo-main-logs/association-hsc-test.db" -d "instrument='HSC' AND skymap='hsc_rings_v1' AND band IN ('g') AND detector<10 AND visit=11690" > hsc-test.out &   nohup pipetask run -b /repo/dc2 -p $AP_PIPE_DIR/pipelines/LsstCamImSim/ApPipeMultiTractFakes.yaml -i u/mrawls/DM- 30802 -AP -o u/mrawls/DM32245-lssttest -c diaPipe:apdb.db_url= "sqlite:////project/mrawls/repo-main-logs/association-lsst-test.db" -d "instrument='LSSTCam-imSim' AND skymap='DC2' AND detector<100 AND exposure=214433" --register-dataset-types > lsst-test.out & Finally, I made an analysis notebook for the full run.

            This is ready for review, thanks for agreeing to do it krzys. There are some pipelines updates in ap_pipe, as well as minor plotting script updates and a new analysis notebook in ap_pipe-notebooks.

            You are also welcome to peruse the outputs in the APDB or the output collection (the exact name of each are in the previous comment).

            mrawls Meredith Rawls added a comment - This is ready for review, thanks for agreeing to do it krzys . There are some pipelines updates in ap_pipe, as well as minor plotting script updates and a new analysis notebook in ap_pipe-notebooks. You are also welcome to peruse the outputs in the APDB or the output collection (the exact name of each are in the previous comment).

            People

              mrawls Meredith Rawls
              mrawls Meredith Rawls
              Krzysztof Findeisen
              Krzysztof Findeisen, Meredith Rawls
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Jenkins

                  No builds found.