Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-15189

Write initial raw-data ingest system for Gen3 Butler

    XMLWordPrintable

    Details

    • Story Points:
      6
    • Epic Link:
    • Sprint:
      BG3_F18_07, BG3_F18_08
    • Team:
      Data Release Production

      Description

      Write abstract interfaces and at least one concrete implementation (AuxTel) for raw data ingest into the Gen3 Butler.

      I will probably actually start with attempting to ingest raw HSC data, as obs_subaru already implements the existing Gen3 customization hooks, and extend to AuxTel on the same ticket to (start to) ensure that the design doesn't "bake in" a particular camera.

        Attachments

          Issue Links

          relates to

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-15503 Improve raw data handling in gen2convert

          • Done

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-15362 Better functionality for metadata stripping in afw

          • Won't Fix

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-15212 Improved transactions and ingest-with-transfer for datastores

          • Done

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-14408 Implement efficient Exposure component access and slicing

          • Invalid

            Activity

            Ascending order - Click to sort in descending order
            Hide
            Permalink
            jbosch Jim Bosch added a comment -

            Leaving some To-Dos on this ticket that should be addressed on DM-15362.

            Show
            jbosch Jim Bosch added a comment - Leaving some To-Dos on this ticket that should be addressed on DM-15362 .
            Hide
            Permalink
            jbosch Jim Bosch added a comment -

            Finally ready for review.  The high-level logic is in a new Task ABC in obs_base, with a concrete subclass in obs_subaru.  daf_butler has a lot of supporting changes, some of which are fairly significant in their own right (e.g. adding runs and collections to templates, and requiring at least one of them).  ci_hsc changes are trivial updates spurred by module-shuffling in obs_subaru.

            Raw ingest for gen3 is by no means done, but this seems like a good stopping point to go fix up other bits of daf_butler and obs_base before returning to it.

            Show
            jbosch Jim Bosch added a comment - Finally ready for review.  The high-level logic is in a new Task ABC in obs_base, with a concrete subclass in obs_subaru.  daf_butler has a lot of supporting changes, some of which are fairly significant in their own right (e.g. adding runs and collections to templates, and requiring at least one of them).  ci_hsc changes are trivial updates spurred by module-shuffling in obs_subaru. Raw ingest for gen3 is by no means done, but this seems like a good stopping point to go fix up other bits of daf_butler and obs_base before returning to it.
            Hide
            Permalink
            tjenness Tim Jenness added a comment -

            Minor comments on PR.

            Show
            tjenness Tim Jenness added a comment - Minor comments on PR.

              People

              Assignee:
              jbosch Jim Bosch
              Reporter:
              jbosch Jim Bosch
              Reviewers:
              Tim Jenness
              Watchers:
              Jim Bosch, Tim Jenness
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Jenkins

                  No builds found.