Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-12618

Define Storage Classes for Butler

    XMLWordPrintable

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: None
    • Labels:
    • Story Points:
      2
    • Epic Link:
    • Sprint:
      BG3_S18_04, BG3_S18_05
    • Team:
      Data Release Production

      Description

      Define the StorageClasses (see DMTN-056) we expect to support.

      Start with Storages supported by Gen2 Butler, add specializations for major composites.

      Can use "generic" (e.g. pickle, afw::table::io) StorageClasses for objects that have no components and no slicing.

      Output should be documented in agreed authoritative form (from DM-12617).

        Attachments

          Activity

          Hide
          tjenness Tim Jenness added a comment -

          Is this ticket still relevant?

          Show
          tjenness Tim Jenness added a comment - Is this ticket still relevant?
          Hide
          pschella Pim Schellart [X] (Inactive) added a comment -

          Why wouldn't it be? We somehow need to ensure we can store everything the Gen2 Butler can.

          Show
          pschella Pim Schellart [X] (Inactive) added a comment - Why wouldn't it be? We somehow need to ensure we can store everything the Gen2 Butler can.
          Hide
          tjenness Tim Jenness added a comment -

          Ah. I read "Define Storage Classes for Butler" as defining how storage classes work, not defining what storage classes we need.

          Show
          tjenness Tim Jenness added a comment - Ah. I read "Define Storage Classes for Butler" as defining how storage classes work, not defining what storage classes we need.
          Hide
          jbosch Jim Bosch added a comment -

          Starting work on this.  I may spin off some StorageClass work to another ticket, but we need to add a few more to run the ci_hsc integration tests we'd planned.

          Show
          jbosch Jim Bosch added a comment - Starting work on this.  I may spin off some StorageClass work to another ticket, but we need to add a few more to run the ci_hsc integration tests we'd planned.
          Hide
          tjenness Tim Jenness added a comment -

          Is DM-12617 really blocking this?

          Show
          tjenness Tim Jenness added a comment - Is DM-12617 really blocking this?
          Hide
          jbosch Jim Bosch added a comment -

          Good catch; no.  Looks like it may still be worth doing in the future, so I won't WontFix it, but it does not need to be a blocker.

          Show
          jbosch Jim Bosch added a comment - Good catch; no.  Looks like it may still be worth doing in the future, so I won't WontFix it, but it does not need to be a blocker.
          Hide
          jbosch Jim Bosch added a comment - - edited

          This ticket brings us near-complete coverage of the Gen2 DatasetTypes used in ci_hsc in terms of concrete StorageClasses and Formatters.  All we're missing are:

          • metadata DatasetTypes: these are PropertySets/PropertyLists, persisted via Boost.Serialization, and their persistence is tied up with daf_persistence at a level I didn't want to try to unravel now for datasets that are basically write-only. (I'd rather not try to unravel it ever; see RFC-482).
          • Stuff that may be stored in calibration repositories (which are not yet converted at all).
          • Complete coverage of the components of composites (namely Exposure).  This simply hasn't been tested, and I want to defer the kind of testing infrastructure that would be required to another ticket.

          I considered trying to reduce the set of concrete StorageClasses by removing the isinstance check on their pytypes; in practice we have a lot of StorageClasses that map to some Python duck-typing interface that doesn't actually have a base class.  I did not proceed with this right now because I think it'd be a distraction from higher-priority things, but we should think about it in the future.

          Show
          jbosch Jim Bosch added a comment - - edited This ticket brings us near-complete coverage of the Gen2 DatasetTypes used in ci_hsc in terms of concrete StorageClasses and Formatters.  All we're missing are: metadata DatasetTypes: these are PropertySets/PropertyLists, persisted via Boost.Serialization, and their persistence is tied up with daf_persistence at a level I didn't want to try to unravel now for datasets that are basically write-only. (I'd rather not try to unravel it ever; see RFC-482 ). Stuff that may be stored in calibration repositories (which are not yet converted at all). Complete coverage of the components of composites (namely Exposure).  This simply hasn't been tested, and I want to defer the kind of testing infrastructure that would be required to another ticket. I considered trying to reduce the set of concrete StorageClasses by removing the isinstance check on their pytypes; in practice we have a lot of StorageClasses that map to some Python duck-typing interface that doesn't actually have a base class.  I did not proceed with this right now because I think it'd be a distraction from higher-priority things, but we should think about it in the future.
          Hide
          pschella Pim Schellart [X] (Inactive) added a comment -

          Good as is.

          Show
          pschella Pim Schellart [X] (Inactive) added a comment - Good as is.

            People

            Assignee:
            jbosch Jim Bosch
            Reporter:
            pschella Pim Schellart [X] (Inactive)
            Reviewers:
            Pim Schellart [X] (Inactive)
            Watchers:
            Jim Bosch, John Swinbank, Pim Schellart [X] (Inactive), Tim Jenness
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved:

                Jenkins

                No builds found.