Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-33300

Drop connections that aren't needed in cp_verify

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None
    • Story Points:
      1
    • Epic Link:
    • Team:
      Data Release Production
    • Urgent?:
      No

      Description

      Tasks that do not use catalogs should drop those connections.

        Attachments

          Activity

          Ascending order - Click to sort in descending order
          Hide
          Permalink
          czw Christopher Waters added a comment -

          https://ci.lsst.codes/blue/organizations/jenkins/stack-os-matrix/detail/stack-os-matrix/35701/pipeline

          Show
          czw Christopher Waters added a comment - https://ci.lsst.codes/blue/organizations/jenkins/stack-os-matrix/detail/stack-os-matrix/35701/pipeline
          Hide
          Permalink
          plazas Andrés Alejandro Plazas Malagón added a comment -

          The following command was failing at the summit: 

          pipetask --long-log run -p ${CP_VERIFY_DIR}/pipelines/VerifyBias.yaml -b /repo/LSSTComCam -o u/plazas/test.2022JAN18.1 -i LSSTComCam/raw/all -j 8 -i u/ocps/3c5f03ea245443dfa0e3600b56004c82,LSSTComCam/calib,LSSTComCam/calib/u/plazas/daily.2022JAN14.5 --register-dataset-types --output-run u/plazas/test.2022JAN18.1/run -d "instrument='LSSTComCam' AND detector IN (0) AND exposure IN (2022011400088, 2022011400089, 2022011400090, 2022011400091, 2022011400092, 2022011400093, 2022011400094, 2022011400095, 2022011400096, 2022011400097)"

          with 

          + OUTPUT_COLLECTION=u/ocps/78c14331b7204c07adb46e436aa983f0
          		 
          + pipetask --long-log run -p '${CP_VERIFY_DIR}/pipelines/VerifyBias.yaml' -b /repo/LSSTComCam -o u/ocps/78c14331b7204c07adb46e436aa983f0 -i LSSTComCam/raw/all -j 8 -i u/ocps/3c5f03ea245443dfa0e3600b56004c82,LSSTComCam/calib,LSSTComCam/calib/u/plazas/daily.2022JAN14.5 --register-dataset-types --output-run u/ocps/78c14331b7204c07adb46e436aa983f0/run -d 'instrument='\''LSSTComCam'\'' AND detector IN (0) AND exposure IN (2022011400088, 2022011400089, 2022011400090, 2022011400091, 2022011400092, 2022011400093, 2022011400094, 2022011400095, 2022011400096, 2022011400097)'
          		 
          WARNING 2022-01-14T23:12:51.870+00:00 lsst.pipe.base.graphBuilder ()(graphBuilder.py:670) - Dataset type uncorrectedSrc is not registered.
          		 
          WARNING 2022-01-14T23:12:51.871+00:00 lsst.pipe.base.graphBuilder ()(graphBuilder.py:670) - Dataset type src is not registered.
          		 
          WARNING 2022-01-14T23:12:51.877+00:00 py.warnings ()(warnings.py:109) - /opt/lsst/software/stack/stack/miniconda3-py38_4.9.2-0.7.0/Linux64/ctrl_mpexec/g80d878e12a+a22c82610c/python/lsst/ctrl/mpexec/cli/script/qgraph.py:183: UserWarning: QuantumGraph is empty
          		 
            qgraph = f.makeGraph(pipelineObj, args)
          		 
           
          		 
          ERROR 2022-01-14T23:12:51.878+00:00 lsst.daf.butler.cli.utils ()(utils.py:897) - Caught an exception, details are in traceback:
          		 
          Traceback (most recent call last):
          		 
            File "/opt/lsst/software/stack/stack/miniconda3-py38_4.9.2-0.7.0/Linux64/ctrl_mpexec/g80d878e12a+a22c82610c/python/lsst/ctrl/mpexec/cli/cmd/commands.py", line 104, in run
          		 
              qgraph = script.qgraph(pipelineObj=pipeline, **kwargs)
          		 
            File "/opt/lsst/software/stack/stack/miniconda3-py38_4.9.2-0.7.0/Linux64/ctrl_mpexec/g80d878e12a+a22c82610c/python/lsst/ctrl/mpexec/cli/script/qgraph.py", line 186, in qgraph
          		 
              raise RuntimeError("QuantumGraph is empty.")
          		 
          RuntimeError: QuantumGraph is empty.

          With the changes in this ticket, it seems to have finished successfully: 

          pipetask --long-log run -p ${CP_VERIFY_DIR}/pipelines/VerifyBias.yaml -b /repo/LSSTComCam -o u/plazas/test.2022JAN19.1 -i LSSTComCam/raw/all -j 8 -i u/ocps/3c5f03ea245443dfa0e3600b56004c82,LSSTComCam/calib,LSSTComCam/calib/u/plazas/daily.2022JAN14.5 --register-dataset-types --output-run u/plazas/test.2022JAN18.1/run -d "instrument='LSSTComCam' AND detector IN (0) AND exposure IN (2022011400088, 2022011400089, 2022011400090, 2022011400091, 2022011400092, 2022011400093, 2022011400094, 2022011400095, 2022011400096, 2022011400097)"


          INFO 2022-01-19T17:13:51.844+00:00 lsst.ctrl.mpexec.singleQuantumExecutor (verifyRunStats:{instrument: 'LSSTComCam'})(singleQuantumExecutor.py:188) - Execution of task 'verifyRunStats' on quantum {instrument: 'LSSTComCam'} took 0.412 seconds
          		 
          INFO 2022-01-19T17:13:51.930+00:00 lsst.ctrl.mpexec.mpGraphExecutor ()(mpGraphExecutor.py:466) - Executed 31 quanta successfully, 0 failed and 0 remain out of total 31 quanta.

          Show
          plazas Andrés Alejandro Plazas Malagón added a comment - The following command was failing at the summit: pipetask --long-log run -p ${CP_VERIFY_DIR}/pipelines/VerifyBias.yaml -b /repo/LSSTComCam -o u/plazas/test.2022JAN18.1 -i LSSTComCam/raw/all -j 8 -i u/ocps/3c5f03ea245443dfa0e3600b56004c82,LSSTComCam/calib,LSSTComCam/calib/u/plazas/daily.2022JAN14.5 --register-dataset-types --output-run u/plazas/test.2022JAN18.1/run -d "instrument='LSSTComCam' AND detector IN (0) AND exposure IN (2022011400088, 2022011400089, 2022011400090, 2022011400091, 2022011400092, 2022011400093, 2022011400094, 2022011400095, 2022011400096, 2022011400097)" with + OUTPUT_COLLECTION=u/ocps/78c14331b7204c07adb46e436aa983f0 + pipetask --long-log run -p '${CP_VERIFY_DIR}/pipelines/VerifyBias.yaml' -b /repo/LSSTComCam -o u/ocps/78c14331b7204c07adb46e436aa983f0 -i LSSTComCam/raw/all -j 8 -i u/ocps/3c5f03ea245443dfa0e3600b56004c82,LSSTComCam/calib,LSSTComCam/calib/u/plazas/daily.2022JAN14.5 --register-dataset-types --output-run u/ocps/78c14331b7204c07adb46e436aa983f0/run -d 'instrument='\''LSSTComCam'\'' AND detector IN (0) AND exposure IN (2022011400088, 2022011400089, 2022011400090, 2022011400091, 2022011400092, 2022011400093, 2022011400094, 2022011400095, 2022011400096, 2022011400097)' WARNING 2022-01-14T23:12:51.870+00:00 lsst.pipe.base.graphBuilder ()(graphBuilder.py:670) - Dataset type uncorrectedSrc is not registered. WARNING 2022-01-14T23:12:51.871+00:00 lsst.pipe.base.graphBuilder ()(graphBuilder.py:670) - Dataset type src is not registered. WARNING 2022-01-14T23:12:51.877+00:00 py.warnings ()(warnings.py:109) - /opt/lsst/software/stack/stack/miniconda3-py38_4.9.2-0.7.0/Linux64/ctrl_mpexec/g80d878e12a+a22c82610c/python/lsst/ctrl/mpexec/cli/script/qgraph.py:183: UserWarning: QuantumGraph is empty qgraph = f.makeGraph(pipelineObj, args)   ERROR 2022-01-14T23:12:51.878+00:00 lsst.daf.butler.cli.utils ()(utils.py:897) - Caught an exception, details are in traceback: Traceback (most recent call last): File "/opt/lsst/software/stack/stack/miniconda3-py38_4.9.2-0.7.0/Linux64/ctrl_mpexec/g80d878e12a+a22c82610c/python/lsst/ctrl/mpexec/cli/cmd/commands.py", line 104, in run qgraph = script.qgraph(pipelineObj=pipeline, **kwargs) File "/opt/lsst/software/stack/stack/miniconda3-py38_4.9.2-0.7.0/Linux64/ctrl_mpexec/g80d878e12a+a22c82610c/python/lsst/ctrl/mpexec/cli/script/qgraph.py", line 186, in qgraph raise RuntimeError("QuantumGraph is empty.") RuntimeError: QuantumGraph is empty. With the changes in this ticket, it seems to have finished successfully: pipetask --long-log run -p ${CP_VERIFY_DIR}/pipelines/VerifyBias.yaml -b /repo/LSSTComCam -o u/plazas/test.2022JAN19.1 -i LSSTComCam/raw/all -j 8 -i u/ocps/3c5f03ea245443dfa0e3600b56004c82,LSSTComCam/calib,LSSTComCam/calib/u/plazas/daily.2022JAN14.5 --register-dataset-types --output-run u/plazas/test.2022JAN18.1/run -d "instrument='LSSTComCam' AND detector IN (0) AND exposure IN (2022011400088, 2022011400089, 2022011400090, 2022011400091, 2022011400092, 2022011400093, 2022011400094, 2022011400095, 2022011400096, 2022011400097)" INFO 2022-01-19T17:13:51.844+00:00 lsst.ctrl.mpexec.singleQuantumExecutor (verifyRunStats:{instrument: 'LSSTComCam'})(singleQuantumExecutor.py:188) - Execution of task 'verifyRunStats' on quantum {instrument: 'LSSTComCam'} took 0.412 seconds INFO 2022-01-19T17:13:51.930+00:00 lsst.ctrl.mpexec.mpGraphExecutor ()(mpGraphExecutor.py:466) - Executed 31 quanta successfully, 0 failed and 0 remain out of total 31 quanta.
          Hide
          Permalink
          plazas Andrés Alejandro Plazas Malagón added a comment -

          This seems to have fixed the problem I was having with the OCPS calibration script failing the verification step. Thanks! I still need to run the OCPS script again when the camera is available to do so.

          What is the purpose of these catalogs? When should they be used (and how are they generated?), and why is it OK to just not use them when they are not available?

          Show
          plazas Andrés Alejandro Plazas Malagón added a comment - This seems to have fixed the problem I was having with the OCPS calibration script failing the verification step. Thanks! I still need to run the OCPS script again when the camera is available to do so. What is the purpose of these catalogs? When should they be used (and how are they generated?), and why is it OK to just not use them when they are not available?

            People

            Assignee:
            czw Christopher Waters
            Reporter:
            czw Christopher Waters
            Reviewers:
            Andrés Alejandro Plazas Malagón
            Watchers:
            Andrés Alejandro Plazas Malagón, Christopher Waters
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved:

                Jenkins

                No builds found.