Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-15256

Include calibration repositories in Gen2->Gen3 conversion

    XMLWordPrintable

    Details

    • Story Points:
      12
    • Epic Link:
    • Sprint:
      BG3_F18_08, BG3_F18_09, BG3_F18_10
    • Team:
      Data Release Production

      Description

      The Gen2->Gen3 converter code (daf.butler.gen2convert) currently ignores calibration repos.  We want to add each Gen2 Calib repo as a new Collection in Gen3.  The tricky part will be converting from the date-range lookup in the Gen2 calibration registries to the ExposureRanges (integer IDs) used by Gen3.

        Attachments

          Issue Links

          is parent task of

          Bug - A problem which impairs or prevents the functions of the product. DM-16065 ci_hsc does not validate calibration data in converted gen3 repository

          • Invalid

            Activity

            Ascending order - Click to sort in descending order
            No builds found.
            jbosch Jim Bosch created issue -
            jbosch Jim Bosch made changes -
            Field Original Value New Value
            Epic Link DM-14448 [ 80390 ]
            jbosch Jim Bosch made changes -
            Risk Score 0
            jbosch Jim Bosch made changes -
            Sprint BG3_F18_08 [ 772 ]
            Hide
            Permalink
            jbosch Jim Bosch added a comment -

            Christopher Waters, this is indeed in progress, right?  (I just want to get Jira reflecting reality.)

            Show
            jbosch Jim Bosch added a comment - Christopher Waters , this is indeed in progress, right?  (I just want to get Jira reflecting reality.)
            jbosch Jim Bosch made changes -
            Status To Do [ 10001 ] In Progress [ 3 ]
            Hide
            Permalink
            czw Christopher Waters added a comment -

            Yes, this is in progress.  Sorry for not marking it as so;  I'm still getting used to the tracking side of development.

            Show
            czw Christopher Waters added a comment - Yes, this is in progress.  Sorry for not marking it as so;  I'm still getting used to the tracking side of development.
            vaikunth Vaikunth Thukral made changes -
            Sprint BG3_F18_08 [ 772 ] BG3_F18_08, BG3_F18_09 [ 772, 779 ]
            czw Christopher Waters made changes -
            Reviewers Jim Bosch [ jbosch ]
            Status In Progress [ 3 ] In Review [ 10004 ]
            Hide
            Permalink
            jbosch Jim Bosch added a comment -

            Some comments on the daf_butler PR, but overall looks good!

            Since I don't see a ci_hsc branch, I assume the state of testing is that ci_hsc automatically runs this new code, but there is no verification there that the calibration Datasets are appearing in the gen3.sqlite3 there.  Have you done a one-off test of that?  How hard would it be to automate it?

            Show
            jbosch Jim Bosch added a comment - Some comments on the daf_butler PR, but overall looks good! Since I don't see a ci_hsc branch, I assume the state of testing is that ci_hsc automatically runs this new code, but there is no verification there that the calibration Datasets are appearing in the gen3.sqlite3 there.  Have you done a one-off test of that?  How hard would it be to automate it?
            jbosch Jim Bosch made changes -
            Status In Review [ 10004 ] Reviewed [ 10101 ]
            vaikunth Vaikunth Thukral made changes -
            Sprint BG3_F18_08, BG3_F18_09 [ 772, 779 ] BG3_F18_08, BG3_F18_09, BG3_F18_10 [ 772, 779, 797 ]
            Hide
            Permalink
            czw Christopher Waters added a comment -

            I have been running independent validation, but had not looked into the validation code in ci_hsc until today.  I'm unclear on how the dataId is being set, and may push and merge my current changes and add a ticket to fix the validation in ci_hsc independently.

            Show
            czw Christopher Waters added a comment - I have been running independent validation, but had not looked into the validation code in ci_hsc until today.  I'm unclear on how the dataId is being set, and may push and merge my current changes and add a ticket to fix the validation in ci_hsc independently.
            czw Christopher Waters made changes -
            Link This issue is parent task of DM-16065 [ DM-16065 ]
            Hide
            Permalink
            czw Christopher Waters added a comment -

            Complete, although I will need to finish automatic validation of the gen3 calibration products (DM-16065).

            Show
            czw Christopher Waters added a comment - Complete, although I will need to finish automatic validation of the gen3 calibration products ( DM-16065 ).
            czw Christopher Waters made changes -
            Resolution Done [ 10000 ]
            Status Reviewed [ 10101 ] Done [ 10002 ]

              People

              Assignee:
              czw Christopher Waters
              Reporter:
              jbosch Jim Bosch
              Reviewers:
              Jim Bosch
              Watchers:
              Christopher Waters, Jim Bosch
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Jenkins

                  No builds found.