Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-17757

Key error when trying to use pipetask to output pipeline/quantum graph and then read it.

    Details

    • Type: Bug
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      Trying to run Nate's full ci_hsc pipeline demo via Pegasus again after DM-17739  The process is to run the same pipetask command but having it output the QuantumGraph and then our code reads the QuantumGraph.    Our pickle reading code failed, so tried using pipetask itself to read the QuantumGraph and it too fails.    I tried various portions of the full pipeline trying to find a small test to reproduce the problem.    I was able to reproduce the problem with two tasks and then further filtered the input dataset.    But if I go back to letting the first pipetask command just run the pipeline, the pipeline runs successfully.

      qgraph dot file has an empty graph.   Get same error message if building and reading QuantumGraph as well as pipeline

      Environment:

      • ci_hsc repo from w_2019_05
      • lsst_distrib -t w_2019_05

      following Nate and Andy recommendations to use git master for:     
      ctrl_mpexec, daf_butler, demo_hsc_pipelinetask,  meas_base,  pipe_base, pipe_tasks, skymap

      QuantumGraph Commands:

      pipetask -L debug -d "Patch.patch = 69 and Visit.visit = 903986" -b $dbutler/butler.yaml -p lsst.meas.base -p lsst.ip.isr -p lsst.pipe.tasks -i shared/ci_hsc -o fail2 qgraph \
      -t calibrate.CalibrateTask:ct -C ct:$DEMO_HSC_PIPELINETASK_DIR/config/calibrate.py \
      -t makeCoaddTempExp.MakeWarpTask:mwt -C mwt:$DEMO_HSC_PIPELINETASK_DIR/config/makeWarp.py \
      -q demo_qgraph.pickle --pipeline-dot pipetask_pipeline.dot --qgraph-dot pipetask_qgraph.dot
      

      pipetask -L debug qgraph -g demo_qgraph.pickle --show=graph
      

      Pipeline Commands:

      pipetask -L debug -d "Patch.patch = 69 and Visit.visit = 903986" -b $dbutler/butler.yaml -p lsst.meas.base -p lsst.ip.isr -p lsst.pipe.tasks -i shared/ci_hsc -o fail2 build \
      -t calibrate.CalibrateTask:ct -C ct:$DEMO_HSC_PIPELINETASK_DIR/config/calibrate.py \
      -t makeCoaddTempExp.MakeWarpTask:mwt -C mwt:$DEMO_HSC_PIPELINETASK_DIR/config/makeWarp.py \
      -s pipeline.pickle
      

      pipetask -L debug build -p pipeline.pickle --show=pipeline
      

      Error message from qgraph:

      Failed to build graph: RegistryField 'astrometry.matcher.sourceSelector' failed validation: Unknown key 'diaCatalog' in Registry/ConfigChoiceField
      For more information read the Field definition at:
        File pex/config/registry.py:316 (__init__)And the Config definition at:
        File meas/astrom/matchPessimisticB.py:62 (<module>)
      Traceback (most recent call last):
        File "/usr/local/lsst_stack/w_2019_05/stack/miniconda3-4.5.4-fcd27eb/Linux64/pex_config/16.0-10-g9d3e444/python/lsst/pex/config/configChoiceField.py", line 246, in __getitem__
          value = self._dict[k]
      KeyError: 'diaCatalog'
      
      

      Will provide full output as attachment

        Attachments

          Issue Links

            Activity

            Hide
            salnikov Andy Salnikov added a comment -

            I'm working on my other non-LSST project today (and I'm very behind on that project so I better spend few more days on it) so someone with more free time and knowledge of pex_config should probably take over this ticket (there is probably 0 person fitting that description ).

            Show
            salnikov Andy Salnikov added a comment - I'm working on my other non-LSST project today (and I'm very behind on that project so I better spend few more days on it) so someone with more free time and knowledge of pex_config should probably take over this ticket (there is probably 0 person fitting that description ).
            Hide
            jbosch Jim Bosch added a comment -

            I'm asking Nate Lust to take this over.

            Show
            jbosch Jim Bosch added a comment - I'm asking Nate Lust to take this over.
            Hide
            nlust Nate Lust added a comment -

            I think I have a handle on this now, and should be able to get through it tomorrow.

            Show
            nlust Nate Lust added a comment - I think I have a handle on this now, and should be able to get through it tomorrow.
            Hide
            mgower Michelle Gower added a comment - - edited

            I discovered using shared/ci_hsc,$COLLECTION (where $COLLECTION was used in the python ingest command lines results in an empty graph (worked in our demo).   Changed the input collection to Nate's' -i raw','calib',ref/ps1_pv3_3pi_20170110,${COLLECTION}:  and the single quantum run fails with 

            Unknown key 'ext_shapeHSM_HsmShapeBj' in Registry/ConfigChoiceField

            Is this a separate problem that needs a new Jira issue?   Do we need to change how we saved the single quantum qgraph pickle file?  

              File "/usr/local/lsst_stack/w_2019_06/stack/miniconda3-4.5.4-fcd27eb/Linux64/ctrl_mpexec/master-gca31c86dbb+1/bin/pipetask", line 26, in <module>
                sys.exit(CmdLineFwk().parseAndRun())
              File "/usr/local/lsst_stack/w_2019_06/stack/miniconda3-4.5.4-fcd27eb/Linux64/ctrl_mpexec/master-gca31c86dbb+1/python/lsst/ctrl/mpexec/cmdLineFwk.py", line 127, in parseAndRun
                qgraph = self.makeGraph(pipeline, taskFactory, args)
              File "/usr/local/lsst_stack/w_2019_06/stack/miniconda3-4.5.4-fcd27eb/Linux64/ctrl_mpexec/master-gca31c86dbb+1/python/lsst/ctrl/mpexec/cmdLineFwk.py", line 324, in makeGraph
                qgraph = pickle.load(pickleFile)
              File "/home/mgower/gen3work/full_hsc/git/pex_config/python/lsst/pex/config/config.py", line 1361, in unreduceConfig
                config.loadFromStream(stream)
              File "/home/mgower/gen3work/full_hsc/git/pex_config/python/lsst/pex/config/config.py", line 984, in loadFromStream
                exec(stream, {}, local)
              File "<string>", line 2689, in <module>
              File "/home/mgower/gen3work/full_hsc/git/pex_config/python/lsst/pex/config/configChoiceField.py", line 252, in __getitem__
                "Unknown key %r in Registry/ConfigChoiceField" % k)
            lsst.pex.config.config.FieldValidationError: RegistryField 'measurement.plugins' failed validation: Unknown key 'ext_shapeHSM_HsmShapeBj' in Registry/ConfigChoiceField
            For more information read the Field definition at:
              File pex/config/registry.py:316 (__init__)And the Config definition at:
              File meas/base/sfm.py:131 (<module>)
            Traceback (most recent call last):
              File "/home/mgower/gen3work/full_hsc/bin/demo_bps.py", line 123, in <module>
                main(sys.argv[1:])
              File "/home/mgower/gen3work/full_hsc/bin/demo_bps.py", line 115, in main
                create_all_schemas(args, demoGraph, qgnodes)
              File "/home/mgower/gen3work/full_hsc/python/bps_funcs.py", line 201, in create_all_schemas
                create_one_schema(args.schargs.format(qlfn=pnode['pfn']))
              File "/home/mgower/gen3work/full_hsc/python/bps_funcs.py", line 179, in create_one_schema
                subprocess.run(shlex.split(cmdline), check=True)
              File "/usr/local/lsst_stack/w_2019_06/python/miniconda3-4.5.4/envs/lsst-scipipe-fcd27eb/lib/python3.6/subprocess.py", line 418, in run
                output=stdout, stderr=stderr)
            subprocess.CalledProcessError: Command '['pipetask', '-b', '/home/mgower/gen3work/full_hsc/DATA/butler.yaml', '-i', 'raw,calib,ref/ps1_pv3_3pi_20170110,demo_inputs', '-o', 'full10', 'run', '--init-only', '--qgraph', 'full10/input/quantum000331.pickle']' returned non-zero exit status 1.
            
            

            Show
            mgower Michelle Gower added a comment - - edited I discovered using shared/ci_hsc,$COLLECTION (where $COLLECTION was used in the python ingest command lines results in an empty graph (worked in our demo).   Changed the input collection to Nate's' -i raw','calib',ref/ps1_pv3_3pi_20170110,${COLLECTION}:  and the single quantum run fails with  Unknown key 'ext_shapeHSM_HsmShapeBj' in Registry/ConfigChoiceField Is this a separate problem that needs a new Jira issue?   Do we need to change how we saved the single quantum qgraph pickle file?   File "/usr/local/lsst_stack/w_2019_06/stack/miniconda3-4.5.4-fcd27eb/Linux64/ctrl_mpexec/master-gca31c86dbb+1/bin/pipetask" , line 26 , in <module> sys.exit(CmdLineFwk().parseAndRun()) File "/usr/local/lsst_stack/w_2019_06/stack/miniconda3-4.5.4-fcd27eb/Linux64/ctrl_mpexec/master-gca31c86dbb+1/python/lsst/ctrl/mpexec/cmdLineFwk.py" , line 127 , in parseAndRun qgraph = self.makeGraph(pipeline, taskFactory, args) File "/usr/local/lsst_stack/w_2019_06/stack/miniconda3-4.5.4-fcd27eb/Linux64/ctrl_mpexec/master-gca31c86dbb+1/python/lsst/ctrl/mpexec/cmdLineFwk.py" , line 324 , in makeGraph qgraph = pickle.load(pickleFile) File "/home/mgower/gen3work/full_hsc/git/pex_config/python/lsst/pex/config/config.py" , line 1361 , in unreduceConfig config.loadFromStream(stream) File "/home/mgower/gen3work/full_hsc/git/pex_config/python/lsst/pex/config/config.py" , line 984 , in loadFromStream exec(stream, {}, local) File "<string>" , line 2689 , in <module> File "/home/mgower/gen3work/full_hsc/git/pex_config/python/lsst/pex/config/configChoiceField.py" , line 252 , in __getitem__ "Unknown key %r in Registry/ConfigChoiceField" % k) lsst.pex.config.config.FieldValidationError: RegistryField 'measurement.plugins' failed validation: Unknown key 'ext_shapeHSM_HsmShapeBj' in Registry/ConfigChoiceField For more information read the Field definition at: File pex/config/registry.py: 316 (__init__)And the Config definition at: File meas/base/sfm.py: 131 (<module>) Traceback (most recent call last): File "/home/mgower/gen3work/full_hsc/bin/demo_bps.py" , line 123 , in <module> main(sys.argv[ 1 :]) File "/home/mgower/gen3work/full_hsc/bin/demo_bps.py" , line 115 , in main create_all_schemas(args, demoGraph, qgnodes) File "/home/mgower/gen3work/full_hsc/python/bps_funcs.py" , line 201 , in create_all_schemas create_one_schema(args.schargs.format(qlfn=pnode[ 'pfn' ])) File "/home/mgower/gen3work/full_hsc/python/bps_funcs.py" , line 179 , in create_one_schema subprocess.run(shlex.split(cmdline), check=True) File "/usr/local/lsst_stack/w_2019_06/python/miniconda3-4.5.4/envs/lsst-scipipe-fcd27eb/lib/python3.6/subprocess.py" , line 418 , in run output=stdout, stderr=stderr) subprocess.CalledProcessError: Command '[' pipetask ', ' -b ', ' /home/mgower/gen3work/full_hsc/DATA/butler.yaml ', ' -i ', ' raw,calib,ref/ps1_pv3_3pi_20170110,demo_inputs ', ' -o ', ' full10 ', ' run ', ' --init-only ', ' --qgraph ', ' full10/input/quantum000331.pickle ']' returned non-zero exit status 1 .
            Hide
            mgower Michelle Gower added a comment -

            Normal information gathering not working because loading the pickle file results in the error.   Attaching pickle file quantum000331.pickle

            Show
            mgower Michelle Gower added a comment - Normal information gathering not working because loading the pickle file results in the error.   Attaching pickle file  quantum000331.pickle

              People

              • Assignee:
                nlust Nate Lust
                Reporter:
                mgower Michelle Gower
                Reviewers:
                Jim Bosch
                Watchers:
                Andy Salnikov, Jim Bosch, Michelle Gower, Nate Lust, Vaikunth Thukral
              • Votes:
                0 Vote for this issue
                Watchers:
                5 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Summary Panel