Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-17757

Key error when trying to use pipetask to output pipeline/quantum graph and then read it.

    Details

    • Type: Bug
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None
    • Templates:
    • Story Points:
      8
    • Sprint:
      DRP S19-3
    • Team:
      Data Release Production

      Description

      Trying to run Nate's full ci_hsc pipeline demo via Pegasus again after DM-17739  The process is to run the same pipetask command but having it output the QuantumGraph and then our code reads the QuantumGraph.    Our pickle reading code failed, so tried using pipetask itself to read the QuantumGraph and it too fails.    I tried various portions of the full pipeline trying to find a small test to reproduce the problem.    I was able to reproduce the problem with two tasks and then further filtered the input dataset.    But if I go back to letting the first pipetask command just run the pipeline, the pipeline runs successfully.

      qgraph dot file has an empty graph.   Get same error message if building and reading QuantumGraph as well as pipeline

      Environment:
      ci_hsc repo from w_2019_05
      lsst_distrib -t w_2019_05
      following Nate and Andy recommendations to use git master for:     
      ctrl_mpexec, daf_butler, demo_hsc_pipelinetask,  meas_base,  pipe_base, pipe_tasks, skymap

      QuantumGraph Commands:

      pipetask -L debug -d "Patch.patch = 69 and Visit.visit = 903986" -b $dbutler/butler.yaml -p lsst.meas.base -p lsst.ip.isr -p lsst.pipe.tasks -i shared/ci_hsc -o fail2 qgraph \
      -t calibrate.CalibrateTask:ct -C ct:$DEMO_HSC_PIPELINETASK_DIR/config/calibrate.py \
      -t makeCoaddTempExp.MakeWarpTask:mwt -C mwt:$DEMO_HSC_PIPELINETASK_DIR/config/makeWarp.py \
      -q demo_qgraph.pickle --pipeline-dot pipetask_pipeline.dot --qgraph-dot pipetask_qgraph.dot
      

      pipetask -L debug qgraph -g demo_qgraph.pickle --show=graph
      

      Pipeline Commands:

      pipetask -L debug -d "Patch.patch = 69 and Visit.visit = 903986" -b $dbutler/butler.yaml -p lsst.meas.base -p lsst.ip.isr -p lsst.pipe.tasks -i shared/ci_hsc -o fail2 build \
      -t calibrate.CalibrateTask:ct -C ct:$DEMO_HSC_PIPELINETASK_DIR/config/calibrate.py \
      -t makeCoaddTempExp.MakeWarpTask:mwt -C mwt:$DEMO_HSC_PIPELINETASK_DIR/config/makeWarp.py \
      -s pipeline.pickle
      

      pipetask -L debug build -p pipeline.pickle --show=pipeline
      

      Error message from qgraph:

      Failed to build graph: RegistryField 'astrometry.matcher.sourceSelector' failed validation: Unknown key 'diaCatalog' in Registry/ConfigChoiceField
      For more information read the Field definition at:
        File pex/config/registry.py:316 (__init__)And the Config definition at:
        File meas/astrom/matchPessimisticB.py:62 (<module>)
      Traceback (most recent call last):
        File "/usr/local/lsst_stack/w_2019_05/stack/miniconda3-4.5.4-fcd27eb/Linux64/pex_config/16.0-10-g9d3e444/python/lsst/pex/config/configChoiceField.py", line 246, in __getitem__
          value = self._dict[k]
      KeyError: 'diaCatalog'
      
      

      Will provide full output as attachment

        Attachments

          Container Issues

            Issue Links

              Activity

                People

                • Assignee:
                  nlust Nate Lust
                  Reporter:
                  mgower Michelle Gower
                  Reviewers:
                  Jim Bosch
                  Watchers:
                  Andy Salnikov, Jim Bosch, Michelle Gower, Nate Lust, Vaikunth Thukral
                • Votes:
                  0 Vote for this issue
                  Watchers:
                  5 Start watching this issue

                  Dates

                  • Created:
                    Updated:
                    Resolved:

                    Summary Panel