Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-19638

Create parent task/script for bootstrapping Gen3 repos

    Details

      Description

      Building on recent work to enable bootstrapping the Gen3 ci_hsc dataset with minimal reliance on gen2convert (DM-19614, DM-19615, DM-19622, DM-19531, DM-19272) and improvements to gen2convert itself (DM-18023), write a task/script that can be used to similarly bootstrap other Gen3 repos, starting with RC2.

      Steps will include:

      • register instrument(s)
      • add curated calibrations, via Instrument.writeCuratedCalibrations
      • register skymaps(s)
      • ingest raws, via RawIngestTask
      • add Gen2-calib-repo calibrations, refcats, and BrightObjectMasks via explicit, lower-level calls to gen2convert code in daf_butler.

      I do not currently plan to include actually creating the empty repository, which will save this task from having to somehow read in and apply overrides to butler yaml configuration.

        Attachments

          Issue Links

            Activity

            jbosch Jim Bosch created issue -
            jbosch Jim Bosch made changes -
            Field Original Value New Value
            Epic Link DM-16771 [ 235477 ]
            jbosch Jim Bosch made changes -
            Description Building on recent work to enable bootstrapping the Gen3 ci_hsc dataset with minimal reliance on gen2convert (DM-19614, DM-19615, DM-19622, DM-19531, DM-19272) and improvements to gen2convert itself (DM-18023), write a task/script that can be used to similarly bootstrap other Gen3 repos, starting with RC2.

            Steps will include:
             * register instrument(s)
             * add curated calibrations, via Instrument.writeCuratedCalibrations
             * register skymaps(s) - via the new MakeGen3SkyMapTask pf DM-19531
             * ingest raws, via RawIngestTask
             * add Gen2-calib-repo calibrations, refcats, and BrightObjectMasks via explicit, lower-level calls to gen2convert code in daf_butler.

            I do not currently plan to include actually creating the empty repository, which will save this task from having to somehow read in and apply overrides to butler yaml configuration.
            Building on recent work to enable bootstrapping the Gen3 ci_hsc dataset with minimal reliance on gen2convert (DM-19614, DM-19615, DM-19622, DM-19531, DM-19272) and improvements to gen2convert itself (DM-18023), write a task/script that can be used to similarly bootstrap other Gen3 repos, starting with RC2.

            Steps will include:
             * register instrument(s)
             * add curated calibrations, via Instrument.writeCuratedCalibrations
             * register skymaps(s)
             * ingest raws, via RawIngestTask
             * add Gen2-calib-repo calibrations, refcats, and BrightObjectMasks via explicit, lower-level calls to gen2convert code in daf_butler.

            I do not currently plan to include actually creating the empty repository, which will save this task from having to somehow read in and apply overrides to butler yaml configuration.
            jbosch Jim Bosch made changes -
            Status To Do [ 10001 ] In Progress [ 3 ]
            jbosch Jim Bosch made changes -
            Reviewers Hsin-Fang Chiang [ hchiang2 ]
            Status In Progress [ 3 ] In Review [ 10004 ]
            hchiang2 Hsin-Fang Chiang made changes -
            Link This issue relates to DM-19797 [ DM-19797 ]
            hchiang2 Hsin-Fang Chiang made changes -
            Status In Review [ 10004 ] Reviewed [ 10101 ]
            jbosch Jim Bosch made changes -
            Resolution Done [ 10000 ]
            Status Reviewed [ 10101 ] Done [ 10002 ]
            czw Christopher Waters made changes -
            Link This issue contains DM-18027 [ DM-18027 ]
            jbosch Jim Bosch made changes -
            Link This issue relates to DM-19961 [ DM-19961 ]

              People

              • Assignee:
                jbosch Jim Bosch
                Reporter:
                jbosch Jim Bosch
                Reviewers:
                Hsin-Fang Chiang
                Watchers:
                Hsin-Fang Chiang, Jim Bosch
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Summary Panel