Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-32245

Reprocess HiTS AP with fakes and an APDB

    XMLWordPrintable

Details

    • Story
    • Status: Done
    • Resolution: Done
    • None
    • None
    • None
    • 8
    • AP F21-6 (November), AP S22-1 (December)
    • Alert Production
    • No

    Description

      As a followup to DM-30703, reprocess HiTS with multi-tract fakes support, and ensure the diaPipe step actually writes to the APDB.

      Attachments

        Issue Links

          Activity

            No builds found.
            mrawls Meredith Rawls created issue -
            sullivan Ian Sullivan made changes -
            Field Original Value New Value
            Epic Link DM-30503 [ 510162 ]
            Sprint AP F21-5 (October) [ 1119 ]
            mrawls Meredith Rawls made changes -
            Link This issue is blocked by DM-31769 [ DM-31769 ]
            sullivan Ian Sullivan made changes -
            Sprint AP F21-5 (October) [ 1119 ] AP F21-6 (November) [ 1124 ]
            sullivan Ian Sullivan made changes -
            Rank Ranked higher
            mrawls Meredith Rawls made changes -
            Status To Do [ 10001 ] In Progress [ 3 ]

            There are four precursor collections, run as separate tracts:

            As reported on DM-32246, they basically ran fine, except whichever overlapping tract ran second had many (mostly expected) diaPipe failures due to duplicate DiaObjects keys.

            I will pick up on this ticket once DM-31964 is complete.

            mrawls Meredith Rawls added a comment - There are four precursor collections, run as separate tracts: u/mrawls/ DM-32245 -DECam-fakes-9812 u/mrawls/ DM-32245 -DECam-fakes-9813 u/mrawls/ DM-32245 -DECam-fakes-8604 u/mrawls/ DM-32245 -DECam-fakes-8605 As reported on DM-32246 , they basically ran fine, except whichever overlapping tract ran second had many (mostly expected) diaPipe failures due to duplicate DiaObjects keys. I will pick up on this ticket once DM-31964 is complete.
            mrawls Meredith Rawls made changes -
            Link This issue is blocked by DM-31964 [ DM-31964 ]

            I did a test run with the DM-31964 branch today on tracts 9812 and 9813. I had to rename (actually "relabel") the tasks originally referred to as visitFakes and matchFakes in the pipeline yaml since the dimensions changed. After the quantum graph took its time building, I got this error:

            (lsst-scipipe) [mrawls@lsst-devl03 repo-main-logs]$ more /project/mrawls/repo-main-logs/bps/u/mrawls/DM-31964-DECam-fakes/20211124T201946Z/quantumGraphGeneration.out
            /software/lsstsw/stack_20210813/stack/miniconda3-py38_4.9.2-0.7.0/Linux64/ctrl_mpexec/22.0.1-29-gedb21c8+82be1abe8f/bin/pipetask qgraph -d "instrument='DECam' AND skymap='hsc_rings_v1' AND 
            band IN ('g') AND exposure>410914 AND exposure<421607 AND tract IN (9812, 9813)" -b /repo/main/butler.yaml -i DECam/defaults/hits2015,DECam/calib,u/mrawls/DM-30703-hits-crosstalk-sources,u/
            mrawls/DM-30703-HiTS2014-templates -p /project/mrawls/repo-main-logs/ApPipe_DM32245.yaml -q /project/mrawls/repo-main-logs/bps/u/mrawls/DM-31964-DECam-fakes/20211124T201946Z/u_mrawls_DM-319
            64-DECam-fakes_20211124T201946Z.qgraph
             
             
            lsst.daf.butler.cli.utils ERROR: Caught an exception, details are in traceback:
            Traceback (most recent call last):
              File "/software/lsstsw/stack_20210813/stack/miniconda3-py38_4.9.2-0.7.0/Linux64/ctrl_mpexec/22.0.1-29-gedb21c8+82be1abe8f/python/lsst/ctrl/mpexec/cli/cmd/commands.py", line 96, in qgraph
                script.qgraph(pipelineObj=pipeline, **kwargs)
              File "/software/lsstsw/stack_20210813/stack/miniconda3-py38_4.9.2-0.7.0/Linux64/ctrl_mpexec/22.0.1-29-gedb21c8+82be1abe8f/python/lsst/ctrl/mpexec/cli/script/qgraph.py", line 156, in qgrap
            h
                qgraph = f.makeGraph(pipelineObj, args)
              File "/software/lsstsw/stack_20210813/stack/miniconda3-py38_4.9.2-0.7.0/Linux64/ctrl_mpexec/22.0.1-29-gedb21c8+82be1abe8f/python/lsst/ctrl/mpexec/cmdLineFwk.py", line 571, in makeGraph
                qgraph = graphBuilder.makeGraph(pipeline, collections, run, args.data_query, metadata=metadata,
              File "/software/lsstsw/stack_20210813/stack/miniconda3-py38_4.9.2-0.7.0/Linux64/pipe_base/22.0.1-32-g19dc4ec+303e5401d2/python/lsst/pipe/base/graphBuilder.py", line 1043, in makeGraph
                return scaffolding.makeQuantumGraph(metadata=metadata)
              File "/software/lsstsw/stack_20210813/stack/miniconda3-py38_4.9.2-0.7.0/Linux64/pipe_base/22.0.1-32-g19dc4ec+303e5401d2/python/lsst/pipe/base/graphBuilder.py", line 929, in makeQuantumGra
            ph
                graph = QuantumGraph(graphInput, metadata=metadata, pruneRefs=self.unfoundRefs)
              File "/software/lsstsw/stack_20210813/stack/miniconda3-py38_4.9.2-0.7.0/Linux64/pipe_base/22.0.1-32-g19dc4ec+303e5401d2/python/lsst/pipe/base/graph/graph.py", line 114, in __init__
                self._buildGraphs(quanta, metadata=metadata, pruneRefs=pruneRefs)
              File "/software/lsstsw/stack_20210813/stack/miniconda3-py38_4.9.2-0.7.0/Linux64/pipe_base/22.0.1-32-g19dc4ec+303e5401d2/python/lsst/pipe/base/graph/graph.py", line 184, in _buildGraphs
                self._datasetRefDict.addProducer(dsRef, value)
              File "/software/lsstsw/stack_20210813/stack/miniconda3-py38_4.9.2-0.7.0/Linux64/pipe_base/22.0.1-32-g19dc4ec+303e5401d2/python/lsst/pipe/base/graph/_implDetails.py", line 82, in addProduc
            er
                raise ValueError(f"Only one node is allowed to produce {key}, "
            ValueError: Only one node is allowed to produce fakes_src@{instrument: 'DECam', detector: 51, visit: 412504, ...}, sc=SourceCatalog], the current producer is QuantumNode(quantum=Quantum(tas
            kName=lsst.pipe.tasks.processCcdWithFakes.ProcessCcdWithFakesTask, dataId={instrument: 'DECam', skymap: 'hsc_rings_v1', detector: 51, tract: 9812, visit: 412504, ...}), taskDef=TaskDef(Proc
            essCcdWithFakesTask, label=processVisitFakes), nodeId=NodeId(number=5084, buildId='1637786834.584632-3801835'))
            

            mrawls Meredith Rawls added a comment - I did a test run with the DM-31964 branch today on tracts 9812 and 9813. I had to rename (actually "relabel") the tasks originally referred to as visitFakes and matchFakes in the pipeline yaml since the dimensions changed. After the quantum graph took its time building, I got this error: (lsst-scipipe) [mrawls@lsst-devl03 repo-main-logs]$ more /project/mrawls/repo-main-logs/bps/u/mrawls/DM-31964-DECam-fakes/20211124T201946Z/quantumGraphGeneration.out /software/lsstsw/stack_20210813/stack/miniconda3-py38_4.9.2-0.7.0/Linux64/ctrl_mpexec/22.0.1-29-gedb21c8+82be1abe8f/bin/pipetask qgraph -d "instrument='DECam' AND skymap='hsc_rings_v1' AND band IN ('g') AND exposure>410914 AND exposure<421607 AND tract IN (9812, 9813)" -b /repo/main/butler.yaml -i DECam/defaults/hits2015,DECam/calib,u/mrawls/DM-30703-hits-crosstalk-sources,u/ mrawls/DM-30703-HiTS2014-templates -p /project/mrawls/repo-main-logs/ApPipe_DM32245.yaml -q /project/mrawls/repo-main-logs/bps/u/mrawls/DM-31964-DECam-fakes/20211124T201946Z/u_mrawls_DM-319 64-DECam-fakes_20211124T201946Z.qgraph     lsst.daf.butler.cli.utils ERROR: Caught an exception, details are in traceback: Traceback (most recent call last): File "/software/lsstsw/stack_20210813/stack/miniconda3-py38_4.9.2-0.7.0/Linux64/ctrl_mpexec/22.0.1-29-gedb21c8+82be1abe8f/python/lsst/ctrl/mpexec/cli/cmd/commands.py", line 96, in qgraph script.qgraph(pipelineObj=pipeline, **kwargs) File "/software/lsstsw/stack_20210813/stack/miniconda3-py38_4.9.2-0.7.0/Linux64/ctrl_mpexec/22.0.1-29-gedb21c8+82be1abe8f/python/lsst/ctrl/mpexec/cli/script/qgraph.py", line 156, in qgrap h qgraph = f.makeGraph(pipelineObj, args) File "/software/lsstsw/stack_20210813/stack/miniconda3-py38_4.9.2-0.7.0/Linux64/ctrl_mpexec/22.0.1-29-gedb21c8+82be1abe8f/python/lsst/ctrl/mpexec/cmdLineFwk.py", line 571, in makeGraph qgraph = graphBuilder.makeGraph(pipeline, collections, run, args.data_query, metadata=metadata, File "/software/lsstsw/stack_20210813/stack/miniconda3-py38_4.9.2-0.7.0/Linux64/pipe_base/22.0.1-32-g19dc4ec+303e5401d2/python/lsst/pipe/base/graphBuilder.py", line 1043, in makeGraph return scaffolding.makeQuantumGraph(metadata=metadata) File "/software/lsstsw/stack_20210813/stack/miniconda3-py38_4.9.2-0.7.0/Linux64/pipe_base/22.0.1-32-g19dc4ec+303e5401d2/python/lsst/pipe/base/graphBuilder.py", line 929, in makeQuantumGra ph graph = QuantumGraph(graphInput, metadata=metadata, pruneRefs=self.unfoundRefs) File "/software/lsstsw/stack_20210813/stack/miniconda3-py38_4.9.2-0.7.0/Linux64/pipe_base/22.0.1-32-g19dc4ec+303e5401d2/python/lsst/pipe/base/graph/graph.py", line 114, in __init__ self._buildGraphs(quanta, metadata=metadata, pruneRefs=pruneRefs) File "/software/lsstsw/stack_20210813/stack/miniconda3-py38_4.9.2-0.7.0/Linux64/pipe_base/22.0.1-32-g19dc4ec+303e5401d2/python/lsst/pipe/base/graph/graph.py", line 184, in _buildGraphs self._datasetRefDict.addProducer(dsRef, value) File "/software/lsstsw/stack_20210813/stack/miniconda3-py38_4.9.2-0.7.0/Linux64/pipe_base/22.0.1-32-g19dc4ec+303e5401d2/python/lsst/pipe/base/graph/_implDetails.py", line 82, in addProduc er raise ValueError(f"Only one node is allowed to produce {key}, " ValueError: Only one node is allowed to produce fakes_src@{instrument: 'DECam', detector: 51, visit: 412504, ...}, sc=SourceCatalog], the current producer is QuantumNode(quantum=Quantum(tas kName=lsst.pipe.tasks.processCcdWithFakes.ProcessCcdWithFakesTask, dataId={instrument: 'DECam', skymap: 'hsc_rings_v1', detector: 51, tract: 9812, visit: 412504, ...}), taskDef=TaskDef(Proc essCcdWithFakesTask, label=processVisitFakes), nodeId=NodeId(number=5084, buildId='1637786834.584632-3801835'))
            sullivan Ian Sullivan made changes -
            Epic Link DM-30503 [ 510162 ] DM-30504 [ 510170 ]
            sullivan Ian Sullivan made changes -
            Story Points 4 8
            mrawls Meredith Rawls added a comment - - edited

            I got all the tracts to process with the pipe_tasks ticket branch of DM-31964! The results are in /repo/main in collection u/mrawls/DM-32245-DECam-fakes-all. The APDB is in postgres lsstdevapdb1, schema mrawls_DM32245.

            I created new ApPipeMultiTractFakes.yaml pipelines in ap_pipe, and tested they work for each of DECam, HSC, and LSSTCam-imSim with w_2021_49 and the pipe_tasks DM-31964 ticket branch. (I realize creating fakes in DC2 is close to pointless, but this served as an integration test for the multi-tract image differencing workflow as well as fakes.)

            These are the commands I ran for integration testing the new pipelines (all were run from /project/mrawls/repo-main-logs):

            nohup pipetask run -b /repo/main -p AP_PIPE_DIR/pipelines/DarkEnergyCamera/ApPipeMultiTractFakes.yaml -i DECam/defaults/hits2015,DECam/calib,u/mrawls/DM-30703-hits-crosstalk-sources,u/mrawls/DM-30703-HiTS2014-templates -o u/mrawls/DM32245-decamtest -c diaPipe:apdb.db_url="sqlite3:////project/mrawls/repo-main-logs/association-decam-test.db" -d "instrument='DECam' AND skymap='hsc_rings_v1' AND band IN ('g') AND detector=51 AND visit=412504" > decam-test.out &
             
            nohup pipetask run -b /repo/main -p $AP_PIPE_DIR/pipelines/HyperSuprimeCam/ApPipeMultiTractFakes.yaml -i u/mrawls/DM-31053-HSC-fakes -o u/mrawls/DM32245-hsctest -c diaPipe:apdb.db_url="sqlite:////project/mrawls/repo-main-logs/association-hsc-test.db" -d "instrument='HSC' AND skymap='hsc_rings_v1' AND band IN ('g') AND detector<10 AND visit=11690" > hsc-test.out &
             
            nohup pipetask run -b /repo/dc2 -p $AP_PIPE_DIR/pipelines/LsstCamImSim/ApPipeMultiTractFakes.yaml -i u/mrawls/DM-30802-AP -o u/mrawls/DM32245-lssttest -c diaPipe:apdb.db_url="sqlite:////project/mrawls/repo-main-logs/association-lsst-test.db" -d "instrument='LSSTCam-imSim' AND skymap='DC2' AND detector<100 AND exposure=214433" --register-dataset-types > lsst-test.out & 

            Finally, I made an analysis notebook for the full run.

            mrawls Meredith Rawls added a comment - - edited I got all the tracts to process with the pipe_tasks ticket branch of DM-31964 ! The results are in /repo/main in collection u/mrawls/ DM-32245 -DECam-fakes-all. The APDB is in postgres lsstdevapdb1, schema mrawls_DM32245. I created new ApPipeMultiTractFakes.yaml pipelines in ap_pipe, and tested they work for each of DECam, HSC, and LSSTCam-imSim with w_2021_49 and the pipe_tasks DM-31964 ticket branch. (I realize creating fakes in DC2 is close to pointless, but this served as an integration test for the multi-tract image differencing workflow as well as fakes.) These are the commands I ran for integration testing the new pipelines (all were run from /project/mrawls/repo-main-logs): nohup pipetask run -b /repo/main -p AP_PIPE_DIR/pipelines/DarkEnergyCamera/ApPipeMultiTractFakes.yaml -i DECam/defaults/hits2015,DECam/calib,u/mrawls/DM- 30703 -hits-crosstalk-sources,u/mrawls/DM- 30703 -HiTS2014-templates -o u/mrawls/DM32245-decamtest -c diaPipe:apdb.db_url= "sqlite3:////project/mrawls/repo-main-logs/association-decam-test.db" -d "instrument='DECam' AND skymap='hsc_rings_v1' AND band IN ('g') AND detector=51 AND visit=412504" > decam-test.out &   nohup pipetask run -b /repo/main -p $AP_PIPE_DIR/pipelines/HyperSuprimeCam/ApPipeMultiTractFakes.yaml -i u/mrawls/DM- 31053 -HSC-fakes -o u/mrawls/DM32245-hsctest -c diaPipe:apdb.db_url= "sqlite:////project/mrawls/repo-main-logs/association-hsc-test.db" -d "instrument='HSC' AND skymap='hsc_rings_v1' AND band IN ('g') AND detector<10 AND visit=11690" > hsc-test.out &   nohup pipetask run -b /repo/dc2 -p $AP_PIPE_DIR/pipelines/LsstCamImSim/ApPipeMultiTractFakes.yaml -i u/mrawls/DM- 30802 -AP -o u/mrawls/DM32245-lssttest -c diaPipe:apdb.db_url= "sqlite:////project/mrawls/repo-main-logs/association-lsst-test.db" -d "instrument='LSSTCam-imSim' AND skymap='DC2' AND detector<100 AND exposure=214433" --register-dataset-types > lsst-test.out & Finally, I made an analysis notebook for the full run.
            mrawls Meredith Rawls made changes -

            This is ready for review, thanks for agreeing to do it krzys. There are some pipelines updates in ap_pipe, as well as minor plotting script updates and a new analysis notebook in ap_pipe-notebooks.

            You are also welcome to peruse the outputs in the APDB or the output collection (the exact name of each are in the previous comment).

            mrawls Meredith Rawls added a comment - This is ready for review, thanks for agreeing to do it krzys . There are some pipelines updates in ap_pipe, as well as minor plotting script updates and a new analysis notebook in ap_pipe-notebooks. You are also welcome to peruse the outputs in the APDB or the output collection (the exact name of each are in the previous comment).
            mrawls Meredith Rawls made changes -
            Reviewers Krzysztof Findeisen [ krzys ]
            Status In Progress [ 3 ] In Review [ 10004 ]
            sullivan Ian Sullivan made changes -
            Sprint AP F21-6 (November) [ 1124 ] AP F21-6 (November), AP S22-1 (December) [ 1124, 1126 ]
            krzys Krzysztof Findeisen made changes -
            Status In Review [ 10004 ] Reviewed [ 10101 ]
            mrawls Meredith Rawls made changes -
            Resolution Done [ 10000 ]
            Status Reviewed [ 10101 ] Done [ 10002 ]
            mrawls Meredith Rawls made changes -
            Link This issue relates to DM-31388 [ DM-31388 ]

            People

              mrawls Meredith Rawls
              mrawls Meredith Rawls
              Krzysztof Findeisen
              Krzysztof Findeisen, Meredith Rawls
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Jenkins

                  No builds found.