Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-32407

Errors concerning a _SpecialTimespanBound when building an execution butler in some cases

    XMLWordPrintable

    Details

    • Story Points:
      1.5
    • Epic Link:
    • Sprint:
      DB_F21_06
    • Team:
      Data Access and Database
    • Urgent?:
      No

      Description

      In certain cases bps submit can fail with an error when building the execution butler. The error is of the form

       

      ctrl.mpexec.cmdLineFwk INFO: QuantumGraph contains 27794 quanta for 7 tasks, graph ID: '1635463022.1406605-3847442'
      		 
      lsst.daf.butler.cli.utils ERROR: Caught an exception, details are in traceback:
      		 
      Traceback (most recent call last):
      		 
        File "/software/lsstsw/stack_20210813/stack/miniconda3-py38_4.9.2-0.7.0/Linux64/ctrl_mpexec/22.0.1-22-gde75665+e965bc080c/python/lsst/ctrl/mpexec/cli/cmd/commands.py", line 96, in qgraph
      		 
          script.qgraph(pipelineObj=pipeline, **kwargs)
      		 
        File "/software/lsstsw/stack_20210813/stack/miniconda3-py38_4.9.2-0.7.0/Linux64/ctrl_mpexec/22.0.1-22-gde75665+e965bc080c/python/lsst/ctrl/mpexec/cli/script/qgraph.py", line 148, in qgraph
      		 
          qgraph = f.makeGraph(pipelineObj, args)
      		 
        File "/software/lsstsw/stack_20210813/stack/miniconda3-py38_4.9.2-0.7.0/Linux64/ctrl_mpexec/22.0.1-22-gde75665+e965bc080c/python/lsst/ctrl/mpexec/cmdLineFwk.py", line 617, in makeGraph
      		 
          buildExecutionButler(butler, qgraph, args.execution_butler_location, run,
      		 
        File "/software/lsstsw/stack_20210813/stack/miniconda3-py38_4.9.2-0.7.0/Linux64/pipe_base/22.0.1-25-g96539fe+64e601228d/python/lsst/pipe/base/executionButlerBuilder.py", line 298, in buildExecutionButler
      		 
          return _import(yamlBuffer, newButler, inserts, run, butlerModifier)
      		 
        File "/software/lsstsw/stack_20210813/stack/miniconda3-py38_4.9.2-0.7.0/Linux64/pipe_base/22.0.1-25-g96539fe+64e601228d/python/lsst/pipe/base/executionButlerBuilder.py", line 197, in _import
      		 
          newButler.import_(filename=yamlBuffer, format="yaml", reuseIds=True, transfer="split")
      		 
        File "/software/lsstsw/stack_20210813/stack/miniconda3-py38_4.9.2-0.7.0/Linux64/daf_butler/22.0.1-79-g8f638725+d18c45d440/python/lsst/daf/butler/_butler.py", line 1794, in import_
      		 
          doImport(filename)
      		 
        File "/software/lsstsw/stack_20210813/stack/miniconda3-py38_4.9.2-0.7.0/Linux64/daf_butler/22.0.1-79-g8f638725+d18c45d440/python/lsst/daf/butler/_butler.py", line 1783, in doImport
      		 
          backend = BackendClass(importStream, self.registry)
      		 
        File "/software/lsstsw/stack_20210813/stack/miniconda3-py38_4.9.2-0.7.0/Linux64/daf_butler/22.0.1-79-g8f638725+d18c45d440/python/lsst/daf/butler/transfers/_yaml.py", line 225, in __init__
      		 
          wrapper = yaml.safe_load(stream)
      		 
        File "/software/lsstsw/stack_20210813/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe/lib/python3.8/site-packages/yaml/__init__.py", line 162, in safe_load
      		 
          return load(stream, SafeLoader)
      		 
        File "/software/lsstsw/stack_20210813/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe/lib/python3.8/site-packages/yaml/__init__.py", line 114, in load
      		 
          return loader.get_single_data()
      		 
        File "/software/lsstsw/stack_20210813/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe/lib/python3.8/site-packages/yaml/constructor.py", line 51, in get_single_data
      		 
          return self.construct_document(node)
      		 
        File "/software/lsstsw/stack_20210813/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe/lib/python3.8/site-packages/yaml/constructor.py", line 60, in construct_document
      		 
          for dummy in generator:
      		 
        File "/software/lsstsw/stack_20210813/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe/lib/python3.8/site-packages/yaml/constructor.py", line 413, in construct_yaml_map
      		 
          value = self.construct_mapping(node)
      		 
        File "/software/lsstsw/stack_20210813/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe/lib/python3.8/site-packages/yaml/constructor.py", line 218, in construct_mapping
      		 
          return super().construct_mapping(node, deep=deep)
      		 
        File "/software/lsstsw/stack_20210813/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe/lib/python3.8/site-packages/yaml/constructor.py", line 143, in construct_mapping
      		 
          value = self.construct_object(value_node, deep=deep)
      		 
        File "/software/lsstsw/stack_20210813/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe/lib/python3.8/site-packages/yaml/constructor.py", line 100, in construct_object
      		 
          data = constructor(self, node)
      		 
        File "/software/lsstsw/stack_20210813/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe/lib/python3.8/site-packages/yaml/constructor.py", line 427, in construct_undefined
      		 
          raise ConstructorError(None, None,
      		 
      yaml.constructor.ConstructorError: could not determine a constructor for the tag 'tag:yaml.org,2002:python/object/apply:lsst.daf.butler.core.timespan._SpecialTimespanBound'
      		 
        in "<file>", line 75290, column 12
      		 
      /software/lsstsw/stack_20210813/stack/miniconda3-py38_4.9.2-0.7.0/Linux64/ctrl_mpexec/22.0.1-22-gde75665+e965bc080c/bin/pipetask qgraph -b /repo/main/butler.yaml -i DECam/raw/all,DECam/calib/PREOPS-597,DECam/defaults/saha_bulge,DECam/calib/DM-28638/curated/19700101T000000Z,DECam/calib/unbounded -o u/kherner/PREOPS-597_20211026 --output-run u/kherner/PREOPS-597_20211026/20211028T203134Z --save-execution-butler /project/kherner/PREOPS-597_SahaBulge/bps/u/kherner/PREOPS-597_20211026/20211028T203134Z/EXEC_REPO-u_kherner_PREOPS-597_20211026_20211028T203134Z -g /project/kherner/PREOPS-597_SahaBulge/bps/u/kherner/PREOPS-597_20211026/20211028T203134Z/u_kherner_PREOPS-597_20211026_20211028T203134Z.qgraph
      		 
       

      where the line and column numbers of the error are of course workflow-dependent. A Slack thread contains more details. In it, Jim Bosch suggests that YAML serialization may be broken for calibrations with timespans having one a None value for one or both of the stand and end points.

       

        Attachments

          Issue Links

          mentioned in

          Page Loading...

            Activity

            Ascending order - Click to sort in descending order
            12 older comments
            Hide
            Permalink
            jbosch Jim Bosch added a comment -

            That format looks good to me. I assume we map the infinite endpoints to null, because they're None in Python?

            I sort of like the idea of using nanoseconds, because I think in practice the YAML is more for machine consumption than human consumption, but on other hand we should probably be using JSON for machine-consumption anyway (though it's a much bigger project to do that for import/export). But no strong feelings on this.

            Show
            jbosch Jim Bosch added a comment - That format looks good to me. I assume we map the infinite endpoints to null , because they're None in Python? I sort of like the idea of using nanoseconds, because I think in practice the YAML is more for machine consumption than human consumption, but on other hand we should probably be using JSON for machine-consumption anyway (though it's a much bigger project to do that for import/export). But no strong feelings on this.
            Hide
            Permalink
            salnikov Andy Salnikov added a comment -

            Tim Jenness, could you re-review my latest change?

            Show
            salnikov Andy Salnikov added a comment - Tim Jenness , could you re-review my latest change?
            Hide
            Permalink
            salnikov Andy Salnikov added a comment - - edited

            > I assume we map the infinite endpoints to null, because they're None in Python?

            Indeed, Infinite is a None timestamp, and it maps to YAML null.

            Show
            salnikov Andy Salnikov added a comment - - edited > I assume we map the infinite endpoints to null, because they're None in Python? Indeed, Infinite is a None timestamp, and it maps to YAML null .
            Hide
            Permalink
            tjenness Tim Jenness added a comment -

            Looks good to me.

            Show
            tjenness Tim Jenness added a comment - Looks good to me.
            Hide
            Permalink
            salnikov Andy Salnikov added a comment -

            Thanks for review and suggestions! Merged.

            Show
            salnikov Andy Salnikov added a comment - Thanks for review and suggestions! Merged.

              People

              Assignee:
              salnikov Andy Salnikov
              Reporter:
              kherner Kenneth Herner
              Reviewers:
              Tim Jenness
              Watchers:
              Andy Salnikov, Jim Bosch, Kenneth Herner, Tim Jenness
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Jenkins

                  No builds found.