Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-17757

Key error when trying to use pipetask to output pipeline/quantum graph and then read it.

    Details

    • Type: Bug
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None
    • Templates:
    • Story Points:
      8
    • Sprint:
      DRP S19-3
    • Team:
      Data Release Production

      Description

      Trying to run Nate's full ci_hsc pipeline demo via Pegasus again after DM-17739  The process is to run the same pipetask command but having it output the QuantumGraph and then our code reads the QuantumGraph.    Our pickle reading code failed, so tried using pipetask itself to read the QuantumGraph and it too fails.    I tried various portions of the full pipeline trying to find a small test to reproduce the problem.    I was able to reproduce the problem with two tasks and then further filtered the input dataset.    But if I go back to letting the first pipetask command just run the pipeline, the pipeline runs successfully.

      qgraph dot file has an empty graph.   Get same error message if building and reading QuantumGraph as well as pipeline

      Environment:

      • ci_hsc repo from w_2019_05
      • lsst_distrib -t w_2019_05

      following Nate and Andy recommendations to use git master for:     
      ctrl_mpexec, daf_butler, demo_hsc_pipelinetask,  meas_base,  pipe_base, pipe_tasks, skymap

      QuantumGraph Commands:

      pipetask -L debug -d "Patch.patch = 69 and Visit.visit = 903986" -b $dbutler/butler.yaml -p lsst.meas.base -p lsst.ip.isr -p lsst.pipe.tasks -i shared/ci_hsc -o fail2 qgraph \
      -t calibrate.CalibrateTask:ct -C ct:$DEMO_HSC_PIPELINETASK_DIR/config/calibrate.py \
      -t makeCoaddTempExp.MakeWarpTask:mwt -C mwt:$DEMO_HSC_PIPELINETASK_DIR/config/makeWarp.py \
      -q demo_qgraph.pickle --pipeline-dot pipetask_pipeline.dot --qgraph-dot pipetask_qgraph.dot
      

      pipetask -L debug qgraph -g demo_qgraph.pickle --show=graph
      

      Pipeline Commands:

      pipetask -L debug -d "Patch.patch = 69 and Visit.visit = 903986" -b $dbutler/butler.yaml -p lsst.meas.base -p lsst.ip.isr -p lsst.pipe.tasks -i shared/ci_hsc -o fail2 build \
      -t calibrate.CalibrateTask:ct -C ct:$DEMO_HSC_PIPELINETASK_DIR/config/calibrate.py \
      -t makeCoaddTempExp.MakeWarpTask:mwt -C mwt:$DEMO_HSC_PIPELINETASK_DIR/config/makeWarp.py \
      -s pipeline.pickle
      

      pipetask -L debug build -p pipeline.pickle --show=pipeline
      

      Error message from qgraph:

      Failed to build graph: RegistryField 'astrometry.matcher.sourceSelector' failed validation: Unknown key 'diaCatalog' in Registry/ConfigChoiceField
      For more information read the Field definition at:
        File pex/config/registry.py:316 (__init__)And the Config definition at:
        File meas/astrom/matchPessimisticB.py:62 (<module>)
      Traceback (most recent call last):
        File "/usr/local/lsst_stack/w_2019_05/stack/miniconda3-4.5.4-fcd27eb/Linux64/pex_config/16.0-10-g9d3e444/python/lsst/pex/config/configChoiceField.py", line 246, in __getitem__
          value = self._dict[k]
      KeyError: 'diaCatalog'
      
      

      Will provide full output as attachment

        Attachments

        1. pipetask_build_pipeline.txt
          161 kB
          Michelle Gower
        2. pipetask_show_pipeline.txt
          37 kB
          Michelle Gower
        3. quantum000331.pickle
          261 kB
          Michelle Gower

          Issue Links

            Activity

              People

              • Assignee:
                nlust Nate Lust
                Reporter:
                mgower Michelle Gower
                Reviewers:
                Jim Bosch
                Watchers:
                Andy Salnikov, Jim Bosch, Michelle Gower, Nate Lust, Vaikunth Thukral
              • Votes:
                0 Vote for this issue
                Watchers:
                5 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Summary Panel