Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-25448

Add skyMap support to butler export/import

    XMLWordPrintable

    Details

    • Team:
      Architecture
    • Urgent?:
      No

      Description

      If you try to use butler export with a deepCoadd_skyMap then when you do the butler import you get the following error:

      LookupError: Could not fetch record for required dimension skymap via keys {'skymap': 'hsc_rings_v1'}.
      

      According to Jim Bosch this is related to https://github.com/lsst/daf_butler/blob/master/python/lsst/daf/butler/core/repoTransfers.py#L334

        Attachments

          Issue Links

            Activity

            Hide
            jbosch Jim Bosch added a comment -

            Proximate cause of the error above this was passing elements=() to saveDatasets instead of leaving that off. But a deeper problem - the TODO line mentioned above - would have become a problem once that was fixed. This ticket should probably try to address all TODO items in repoTransfers.py; I think the issues that made those hard to address originally have now been resolved, and import/export are now becoming quite popular.

            Show
            jbosch Jim Bosch added a comment - Proximate cause of the error above this was passing elements=() to saveDatasets instead of leaving that off. But a deeper problem - the TODO line mentioned above - would have become a problem once that was fixed. This ticket should probably try to address all TODO items in repoTransfers.py; I think the issues that made those hard to address originally have now been resolved, and import/export are now becoming quite popular.
            Hide
            krzys Krzysztof Findeisen added a comment - - edited

            EDIT: moved to DM-25786.

            Show
            krzys Krzysztof Findeisen added a comment - - edited EDIT: moved to DM-25786 .
            Hide
            tjenness Tim Jenness added a comment -

            The problem with sphgeom is caused by import using safe_load but export dumping sphgeom objects that haven't got a default serializer registered with yaml. YAML will not import arbitrary python objects using safe_load so this can only be fixed by switching to the unsafe loader (not a good idea) or registering proper yaml serialization callbacks in sphgeom package itself.

            Show
            tjenness Tim Jenness added a comment - The problem with sphgeom is caused by import using safe_load but export dumping sphgeom objects that haven't got a default serializer registered with yaml. YAML will not import arbitrary python objects using safe_load so this can only be fixed by switching to the unsafe loader (not a good idea) or registering proper yaml serialization callbacks in sphgeom package itself.
            Hide
            jbosch Jim Bosch added a comment - - edited

            I think this was actually fully fixed by DM-25786 (along with the usage change from my first comment, above). Eli Rykoff, is the original exercise that triggered this something you'd like to try again (when you have time)?

            Show
            jbosch Jim Bosch added a comment - - edited I think this was actually fully fixed by DM-25786 (along with the usage change from my first comment, above). Eli Rykoff , is the original exercise that triggered this something you'd like to try again (when you have time)?

              People

              Assignee:
              Unassigned Unassigned
              Reporter:
              erykoff Eli Rykoff
              Watchers:
              Eli Rykoff, Jim Bosch, Tim Jenness
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

                Dates

                Created:
                Updated:

                  CI Builds

                  No builds found.