Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-11033

Problem with exists/getStorage() with composite datatypes

    Details

    • Type: Bug
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: butler
    • Labels:
      None

      Description

      When using composite data types (e.g. with obs_comCam), seemingly all cmdLineTasks (certainly processCcd.py and constructCalibs.py) fail with

      File "/ssd/lsstsw/stack_20170409/Linux64/daf_persistence/13.0-17-gd5d205a+2/python/lsst/daf/persistence/repository.py", line 303, in exists
          butlerLocationStorage = location.getStorage()
      AttributeError: 'ButlerComposite' object has no attribute 'getStorage'
      

      It seems that these composite objects don't have the function defined.

      Steps to reproduce can be found by setting up a stack on lsst-dev with . /home/mfl/nate.sh and the repo can be remade and the failing command run with . /home/mfl/rerun_nate.sh.

        Attachments

          Issue Links

            Activity

            Hide
            rhl Robert Lupton added a comment - - edited

            Note that `butler.get` succeeds on these data, so there is a non-common code path.

            I have worked around similar problems with e.g. std_raw failing on composites by providing an raw non-composite type with a template, and making std_raw call std_raw (two _ s). I'd like to avoid having to do this, and it's possible that this ticket could be solved at the same time.

            Show
            rhl Robert Lupton added a comment - - edited Note that `butler.get` succeeds on these data, so there is a non-common code path. I have worked around similar problems with e.g. std_raw failing on composites by providing an raw non-composite type with a template, and making std_raw call std _raw (two _ s). I'd like to avoid having to do this, and it's possible that this ticket could be solved at the same time.
            Hide
            rhl Robert Lupton added a comment -

            A more complete traceback is:

              File "/ssd/lsstsw/stack/Linux64/ctrl_pool/13.0-5-g9cf35e0+4/python/lsst/ctrl/pool/parallel.py", line 333, in parse_args
                args.parent = self._parent.parse_args(config, args=leftover, **kwargs)
              File "/ssd/lsstsw/stack/Linux64/pipe_drivers/13.0-7-g11ca5b2+4/python/lsst/pipe/drivers/constructCalibs.py", line 269, in parse_args
                namespace = ArgumentParser.parse_args(self, *args, **kwargs)
              File "/home/rlupton/LSST/pipe/base/python/lsst/pipe/base/argumentParser.py", line 521, in parse_args
                self._processDataIds(namespace)
              File "/home/rlupton/LSST/pipe/base/python/lsst/pipe/base/argumentParser.py", line 626, in _processDataIds
                dataIdContainer.makeDataRefList(namespace)
              File "/home/rlupton/LSST/pipe/base/python/lsst/pipe/base/argumentParser.py", line 141, in makeDataRefList
                dataRef=dr)]
              File "/home/rlupton/LSST/pipe/base/python/lsst/pipe/base/argumentParser.py", line 1021, in dataExists
                return butler.datasetExists(datasetType=datasetType, dataId=dataRef.dataId)
              File "/home/rlupton/LSST/daf/persistence/python/lsst/daf/persistence/butler.py", line 822, in datasetExists
                for _datasetType in location.componentInfo.keys():
              File "/home/rlupton/LSST/daf/persistence/python/lsst/daf/persistence/repository.py", line 303, in exists
                butlerLocationStorage = location.getStorage()
            AttributeError: 'ButlerComposite' object has no attribute 'getStorage'
            

            So that the problem is calling datasetExists. Note that (as reported on slack) I have pushed a hack-around to daf_persistence branch u/rhl/DM-11033

            Show
            rhl Robert Lupton added a comment - A more complete traceback is: File "/ssd/lsstsw/stack/Linux64/ctrl_pool/13.0-5-g9cf35e0+4/python/lsst/ctrl/pool/parallel.py", line 333, in parse_args args.parent = self._parent.parse_args(config, args=leftover, **kwargs) File "/ssd/lsstsw/stack/Linux64/pipe_drivers/13.0-7-g11ca5b2+4/python/lsst/pipe/drivers/constructCalibs.py", line 269, in parse_args namespace = ArgumentParser.parse_args(self, *args, **kwargs) File "/home/rlupton/LSST/pipe/base/python/lsst/pipe/base/argumentParser.py", line 521, in parse_args self._processDataIds(namespace) File "/home/rlupton/LSST/pipe/base/python/lsst/pipe/base/argumentParser.py", line 626, in _processDataIds dataIdContainer.makeDataRefList(namespace) File "/home/rlupton/LSST/pipe/base/python/lsst/pipe/base/argumentParser.py", line 141, in makeDataRefList dataRef=dr)] File "/home/rlupton/LSST/pipe/base/python/lsst/pipe/base/argumentParser.py", line 1021, in dataExists return butler.datasetExists(datasetType=datasetType, dataId=dataRef.dataId) File "/home/rlupton/LSST/daf/persistence/python/lsst/daf/persistence/butler.py", line 822, in datasetExists for _datasetType in location.componentInfo.keys(): File "/home/rlupton/LSST/daf/persistence/python/lsst/daf/persistence/repository.py", line 303, in exists butlerLocationStorage = location.getStorage() AttributeError: 'ButlerComposite' object has no attribute 'getStorage' So that the problem is calling datasetExists . Note that (as reported on slack) I have pushed a hack-around to daf_persistence branch u/rhl/ DM-11033
            Hide
            rhl Robert Lupton added a comment -

            Jim Bosch: Does the superTask design handle this correctly, including the cases that Russell Owen mentions (some data fails to process)?

            Show
            rhl Robert Lupton added a comment - Jim Bosch : Does the superTask design handle this correctly, including the cases that Russell Owen mentions (some data fails to process)?
            Hide
            jbosch Jim Bosch added a comment -

            As far as I can tell from this ticket, I think this issue is currently well below the level of detail of the design sketch for SuperTask-inspired/demanded Butler changes (and probably orthogonal to it). But I don't see the Russell Owen comment you're referring to, so it's possible it's more connected to those changes than I think.

            Show
            jbosch Jim Bosch added a comment - As far as I can tell from this ticket, I think this issue is currently well below the level of detail of the design sketch for SuperTask-inspired/demanded Butler changes (and probably orthogonal to it). But I don't see the Russell Owen comment you're referring to, so it's possible it's more connected to those changes than I think.
            Hide
            rhl Robert Lupton added a comment -

            Sorry, wrong ticket.

            Show
            rhl Robert Lupton added a comment - Sorry, wrong ticket.
            Hide
            npease Nate Pease added a comment -
            Show
            npease Nate Pease added a comment - the obs_base PR is at https://github.com/lsst/obs_base/pull/48
            Hide
            kennylo Kenny Lo added a comment -

            Looking good to merge.

            Show
            kennylo Kenny Lo added a comment - Looking good to merge.

              People

              • Assignee:
                npease Nate Pease
                Reporter:
                mfisherlevine Merlin Fisher-Levine
                Reviewers:
                Kenny Lo
                Watchers:
                Jim Bosch, Kenny Lo, Merlin Fisher-Levine, Nate Pease, Robert Lupton
              • Votes:
                0 Vote for this issue
                Watchers:
                5 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Summary Panel