Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-17757

Key error when trying to use pipetask to output pipeline/quantum graph and then read it.


    • Type: Bug
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: None
    • Labels:
    • Templates:
    • Story Points:
    • Sprint:
      DRP S19-3
    • Team:
      Data Release Production


      Trying to run Nate's full ci_hsc pipeline demo via Pegasus again after DM-17739  The process is to run the same pipetask command but having it output the QuantumGraph and then our code reads the QuantumGraph.    Our pickle reading code failed, so tried using pipetask itself to read the QuantumGraph and it too fails.    I tried various portions of the full pipeline trying to find a small test to reproduce the problem.    I was able to reproduce the problem with two tasks and then further filtered the input dataset.    But if I go back to letting the first pipetask command just run the pipeline, the pipeline runs successfully.

      qgraph dot file has an empty graph.   Get same error message if building and reading QuantumGraph as well as pipeline


      • ci_hsc repo from w_2019_05
      • lsst_distrib -t w_2019_05

      following Nate and Andy recommendations to use git master for:     
      ctrl_mpexec, daf_butler, demo_hsc_pipelinetask,  meas_base,  pipe_base, pipe_tasks, skymap

      QuantumGraph Commands:

      pipetask -L debug -d "Patch.patch = 69 and Visit.visit = 903986" -b $dbutler/butler.yaml -p lsst.meas.base -p lsst.ip.isr -p lsst.pipe.tasks -i shared/ci_hsc -o fail2 qgraph \
      -t calibrate.CalibrateTask:ct -C ct:$DEMO_HSC_PIPELINETASK_DIR/config/calibrate.py \
      -t makeCoaddTempExp.MakeWarpTask:mwt -C mwt:$DEMO_HSC_PIPELINETASK_DIR/config/makeWarp.py \
      -q demo_qgraph.pickle --pipeline-dot pipetask_pipeline.dot --qgraph-dot pipetask_qgraph.dot

      pipetask -L debug qgraph -g demo_qgraph.pickle --show=graph

      Pipeline Commands:

      pipetask -L debug -d "Patch.patch = 69 and Visit.visit = 903986" -b $dbutler/butler.yaml -p lsst.meas.base -p lsst.ip.isr -p lsst.pipe.tasks -i shared/ci_hsc -o fail2 build \
      -t calibrate.CalibrateTask:ct -C ct:$DEMO_HSC_PIPELINETASK_DIR/config/calibrate.py \
      -t makeCoaddTempExp.MakeWarpTask:mwt -C mwt:$DEMO_HSC_PIPELINETASK_DIR/config/makeWarp.py \
      -s pipeline.pickle

      pipetask -L debug build -p pipeline.pickle --show=pipeline

      Error message from qgraph:

      Failed to build graph: RegistryField 'astrometry.matcher.sourceSelector' failed validation: Unknown key 'diaCatalog' in Registry/ConfigChoiceField
      For more information read the Field definition at:
        File pex/config/registry.py:316 (__init__)And the Config definition at:
        File meas/astrom/matchPessimisticB.py:62 (<module>)
      Traceback (most recent call last):
        File "/usr/local/lsst_stack/w_2019_05/stack/miniconda3-4.5.4-fcd27eb/Linux64/pex_config/16.0-10-g9d3e444/python/lsst/pex/config/configChoiceField.py", line 246, in __getitem__
          value = self._dict[k]
      KeyError: 'diaCatalog'

      Will provide full output as attachment


          Issue Links



              • Assignee:
                nlust Nate Lust
                mgower Michelle Gower
                Jim Bosch
                Andy Salnikov, Jim Bosch, Michelle Gower, Nate Lust, Vaikunth Thukral
              • Votes:
                0 Vote for this issue
                5 Start watching this issue


                • Created:

                  Summary Panel