Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-18829

Initiate the Gen3 HSC-RC2 repo on Oracle

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      Create a Gen3 Butler repo containing HSC-RC2 data (raw, calib, refcat..) based on Oracle registry.

      The goal is to do Gen3 ingestion if possible. Otherwise, bootstrap the Gen3 repo from Gen2 repos (gen2convert).

        Attachments

          Issue Links

            Activity

            Hide
            hchiang2 Hsin-Fang Chiang added a comment -

            Use the tickets/DM-19638 branches for now, and my own oracle wallet credential.

            Show
            hchiang2 Hsin-Fang Chiang added a comment - Use the tickets/ DM-19638 branches for now, and my own oracle wallet credential.
            Hide
            hchiang2 Hsin-Fang Chiang added a comment -

            A bootstrapped repo using tickets/DM-19638 early 5/14 versions is at
            /project/hchiang2/gen3repos/w_2019_19/repo2/

            That job took ~19 hours.

            Will do another bootstrap after the merge of DM-19638 assuming that get included in the next weekly release.

            Show
            hchiang2 Hsin-Fang Chiang added a comment - A bootstrapped repo using tickets/ DM-19638 early 5/14 versions is at /project/hchiang2/gen3repos/w_2019_19/repo2/ That job took ~19 hours. Will do another bootstrap after the merge of DM-19638 assuming that get included in the next weekly release.
            Hide
            hchiang2 Hsin-Fang Chiang added a comment -

            Made another RC2 Butler repo using w_2019_20 and Oracle; the job took ~18 hour. Chris did an export. The repo is at /project/hchiang2/gen3repos/w_2019_20/repo

            Show
            hchiang2 Hsin-Fang Chiang added a comment - Made another RC2 Butler repo using w_2019_20 and Oracle; the job took ~18 hour. Chris did an export. The repo is at /project/hchiang2/gen3repos/w_2019_20/repo
            Hide
            hchiang2 Hsin-Fang Chiang added a comment -

            Updated calibration: 

             UPDATE calibration_label SET valid_last = valid_last + 1 WHERE calibration_label != 'unbounded';

            Show
            hchiang2 Hsin-Fang Chiang added a comment - Updated calibration:    UPDATE calibration_label SET valid_last = valid_last + 1 WHERE calibration_label != 'unbounded';
            Hide
            hchiang2 Hsin-Fang Chiang added a comment -

            Without the above correction, errors related to calibration was seen such as
            {{lsst.pipe.base.graphBuilder.PrerequisiteMissingError: Deferred search failed for prerequisite dataset dark using data ID

            {physical_filter: HSC-I, exposure: 19682, visit: 19682, instrument: HSC, detector: 22, abstract_filter: i}

            .}}

            Show
            hchiang2 Hsin-Fang Chiang added a comment - Without the above correction, errors related to calibration was seen such as {{lsst.pipe.base.graphBuilder.PrerequisiteMissingError: Deferred search failed for prerequisite dataset dark using data ID {physical_filter: HSC-I, exposure: 19682, visit: 19682, instrument: HSC, detector: 22, abstract_filter: i} .}}
            Hide
            hchiang2 Hsin-Fang Chiang added a comment -

            The said repo (/project/hchiang2/gen3repos/w_2019_20/repo) was used in the runs for the DMLT milestone report on Jun 4, and will be used to finish the current Gen3 epic

            Show
            hchiang2 Hsin-Fang Chiang added a comment - The said repo (/project/hchiang2/gen3repos/w_2019_20/repo) was used in the runs for the DMLT milestone report on Jun 4, and will be used to finish the current Gen3 epic

              People

              • Assignee:
                hchiang2 Hsin-Fang Chiang
                Reporter:
                hchiang2 Hsin-Fang Chiang
                Watchers:
                Christopher Stephens, Hsin-Fang Chiang, Michelle Gower
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: