Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-15189

Write initial raw-data ingest system for Gen3 Butler

    XMLWordPrintable

    Details

    • Story Points:
      6
    • Epic Link:
    • Sprint:
      BG3_F18_07, BG3_F18_08
    • Team:
      Data Release Production

      Description

      Write abstract interfaces and at least one concrete implementation (AuxTel) for raw data ingest into the Gen3 Butler.

      I will probably actually start with attempting to ingest raw HSC data, as obs_subaru already implements the existing Gen3 customization hooks, and extend to AuxTel on the same ticket to (start to) ensure that the design doesn't "bake in" a particular camera.

        Attachments

          Issue Links

          relates to

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-15503 Improve raw data handling in gen2convert

          • Done

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-15362 Better functionality for metadata stripping in afw

          • Won't Fix

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-15212 Improved transactions and ingest-with-transfer for datastores

          • Done

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-14408 Implement efficient Exposure component access and slicing

          • Invalid

            Activity

            Ascending order - Click to sort in descending order
            No builds found.
            jbosch Jim Bosch created issue -
            jbosch Jim Bosch made changes -
            Field Original Value New Value
            Risk Score 1
            jbosch Jim Bosch made changes -
            Status To Do [ 10001 ] In Progress [ 3 ]
            jbosch Jim Bosch made changes -
            Sprint BG3_F18_07 [ 759 ]
            jbosch Jim Bosch made changes -
            Link This issue relates to DM-15212 [ DM-15212 ]
            jbosch Jim Bosch made changes -
            Story Points 8 6
            jbosch Jim Bosch made changes -
            Risk Score 1 0
            vaikunth Vaikunth Thukral made changes -
            Sprint BG3_F18_07 [ 759 ] BG3_F18_07, BG3_F18_08 [ 759, 772 ]
            Hide
            Permalink
            jbosch Jim Bosch added a comment -

            Leaving some To-Dos on this ticket that should be addressed on DM-15362.

            Show
            jbosch Jim Bosch added a comment - Leaving some To-Dos on this ticket that should be addressed on DM-15362 .
            jbosch Jim Bosch made changes -
            Link This issue relates to DM-15362 [ DM-15362 ]
            jbosch Jim Bosch made changes -
            Link This issue relates to DM-15503 [ DM-15503 ]
            Hide
            Permalink
            jbosch Jim Bosch added a comment -

            Finally ready for review.  The high-level logic is in a new Task ABC in obs_base, with a concrete subclass in obs_subaru.  daf_butler has a lot of supporting changes, some of which are fairly significant in their own right (e.g. adding runs and collections to templates, and requiring at least one of them).  ci_hsc changes are trivial updates spurred by module-shuffling in obs_subaru.

            Raw ingest for gen3 is by no means done, but this seems like a good stopping point to go fix up other bits of daf_butler and obs_base before returning to it.

            Show
            jbosch Jim Bosch added a comment - Finally ready for review.  The high-level logic is in a new Task ABC in obs_base, with a concrete subclass in obs_subaru.  daf_butler has a lot of supporting changes, some of which are fairly significant in their own right (e.g. adding runs and collections to templates, and requiring at least one of them).  ci_hsc changes are trivial updates spurred by module-shuffling in obs_subaru. Raw ingest for gen3 is by no means done, but this seems like a good stopping point to go fix up other bits of daf_butler and obs_base before returning to it.
            jbosch Jim Bosch made changes -
            Reviewers Tim Jenness [ tjenness ]
            Status In Progress [ 3 ] In Review [ 10004 ]
            Hide
            Permalink
            tjenness Tim Jenness added a comment -

            Minor comments on PR.

            Show
            tjenness Tim Jenness added a comment - Minor comments on PR.
            tjenness Tim Jenness made changes -
            Status In Review [ 10004 ] Reviewed [ 10101 ]
            jbosch Jim Bosch made changes -
            Resolution Done [ 10000 ]
            Status Reviewed [ 10101 ] Done [ 10002 ]
            jbosch Jim Bosch made changes -
            Epic Link DM-14448 [ 80390 ]
            jbosch Jim Bosch made changes -
            Link This issue relates to DM-14408 [ DM-14408 ]

              People

              Assignee:
              jbosch Jim Bosch
              Reporter:
              jbosch Jim Bosch
              Reviewers:
              Tim Jenness
              Watchers:
              Jim Bosch, Tim Jenness
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Jenkins

                  No builds found.