Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-24432 Add CALIBRATION collections and remove the calibration_label dimension
  3. DM-26629

Convert calibration datasets to use CALIBRATION collections instead of calibration_label

    XMLWordPrintable

    Details

    • Team:
      Data Release Production

      Description

      Previous DM-24432 subtasks will have added all butler-level functionality needed to use CALIBRATION collections, so this ticket is mostly about fixing downstream code (particular QG generation and various DatasetType definitions) before finally removing the calibration_label dimension from daf_butler.

        Attachments

          Issue Links

          blocks

          Improvement - An improvement or enhancement to an existing feature or task. DM-26944 Move certifyCalibrations.py content to a butler command

          • Done
          relates to

          Bug - A problem which impairs or prevents the functions of the product. DM-26953 Update ap_verify system for calibration collections

          • Done

            Activity

            Ascending order - Click to sort in descending order
            24 older comments
            Hide
            Permalink
            krzys Krzysztof Findeisen added a comment -

            I have no objection to merging today; other tickets have kept ap_verify from running in Gen 3 this week anyway.

            Show
            krzys Krzysztof Findeisen added a comment - I have no objection to merging today; other tickets have kept ap_verify from running in Gen 3 this week anyway.
            Hide
            Permalink
            jbosch Jim Bosch added a comment -

            Status report: I think I've got everything but pipelines_check rebased, through review, and passing tests locally.  I'm still waiting on my local ci_hsc_gen3 to finish running, though, to in order to generate the new data for pipelines_check.  And once pipelines_check is done locally, I'll need one more Jenkins run for it all.

            I did just notice Simon Krughoff's comment on DM-26738 about Gen3 breakage to the new validate_drp_gen3 package, and if I can I'd like to test that I haven't broken that as well.  From what I know about it, I don't think I will break it, since it runs gen2to3 conversion and I'm not breaking Gen2 repos, but I'd like to check if I can.  However, I don't actually see it in the lsst or lsst-dm GitHub orgs.  Anyone know where I can find it, or better, how I can run it through Jenkins?

            Show
            jbosch Jim Bosch added a comment - Status report: I think I've got everything but pipelines_check rebased, through review, and passing tests locally.  I'm still waiting on my local ci_hsc_gen3 to finish running, though, to in order to generate the new data for pipelines_check.  And once pipelines_check is done locally, I'll need one more Jenkins run for it all. I did just notice Simon Krughoff 's comment on DM-26738 about Gen3 breakage to the new  validate_drp_gen3 package, and if I can I'd like to test that I haven't broken that as well.  From what I know about it, I don't think I will break it, since it runs gen2to3 conversion and I'm not breaking Gen2 repos, but I'd like to check if I can.  However, I don't actually see it in the lsst or lsst-dm GitHub orgs.  Anyone know where I can find it, or better, how I can run it through Jenkins?
            Hide
            Permalink
            jbosch Jim Bosch added a comment -

            pipelines_check is updated and two Jenkins runs are off to the races (stagged because ci_hsc was ready first and will probably take longer)

            The PR for pipelines_check is here if anyone else is still following along, but I'll merge that without review if all tests pass, and deal with any style concerns later.

            Show
            jbosch Jim Bosch added a comment - pipelines_check is updated and two Jenkins runs are off to the races (stagged because ci_hsc was ready first and will probably take longer) lsst_distrib + lsst_ci: https://ci.lsst.codes/blue/organizations/jenkins/stack-os-matrix/detail/stack-os-matrix/32718/pipeline/ lsst_distrib + ci_hsc: https://ci.lsst.codes/blue/organizations/jenkins/stack-os-matrix/detail/stack-os-matrix/32717/pipeline The PR for pipelines_check is here if anyone else is still following along, but I'll merge that without review if all tests pass, and deal with any style concerns later.
            Hide
            Permalink
            tjenness Tim Jenness added a comment -

            I think https://github.com/lsst-dmsst/metric-pipeline-tasks is the package Simon fixed today.

            Show
            tjenness Tim Jenness added a comment - I think https://github.com/lsst-dmsst/metric-pipeline-tasks is the package Simon fixed today.
            Hide
            Permalink
            jbosch Jim Bosch added a comment -

            Everyone has signed off individually here, so marking this as as Reviewed.

            Show
            jbosch Jim Bosch added a comment - Everyone has signed off individually here, so marking this as as Reviewed.

              People

              Assignee:
              jbosch Jim Bosch
              Reporter:
              jbosch Jim Bosch
              Reviewers:
              Nate Lust, Tim Jenness
              Watchers:
              Christopher Waters, Jim Bosch, Krzysztof Findeisen, Nate Lust, Simon Krughoff, Tim Jenness
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Jenkins

                  No builds found.