Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-35639

Switch AP and DRP pipelines to use new image differencing

    XMLWordPrintable

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: ap_pipe
    • Labels:
      None

      Description

      Switch the AP and DRP pipelines to use the new image differencing Task, with forceCompatibility=True.

        Attachments

          Issue Links

            Activity

            Hide
            sullivan Ian Sullivan added a comment - - edited

            Packages to modify:
            drp_pipe
            ap_pipe
            ap_verify
            obs_base to modify the *Diff_config entries in obs_base/policy/datasets.yaml (not needed following DM-35035)
            pipe_tasks to add deprecation warning
            ip_diffim to set forceCompatibility=True
            ci_hsc_gen3 to set pipelineTask names in final tests
            ci_imsim to update bps memory allocation settings
            verify_metrics to update metric names and sources
            pipelines_lsst_io to update documentation
            meas_transiNet

            Show
            sullivan Ian Sullivan added a comment - - edited Packages to modify: drp_pipe ap_pipe ap_verify obs_base to modify the *Diff_config entries in obs_base/policy/datasets.yaml (not needed following DM-35035 ) pipe_tasks to add deprecation warning ip_diffim to set forceCompatibility=True ci_hsc_gen3 to set pipelineTask names in final tests ci_imsim to update bps memory allocation settings verify_metrics to update metric names and sources pipelines_lsst_io to update documentation meas_transiNet
            Show
            sullivan Ian Sullivan added a comment - - edited Pull requests: https://github.com/lsst/drp_pipe/pull/33 https://github.com/lsst/ip_diffim/pull/227 https://github.com/lsst/ap_pipe/pull/120 https://github.com/lsst/ap_verify/pull/165 https://github.com/lsst/pipe_tasks/pull/705 https://github.com/lsst/verify_metrics/pull/35 https://github.com/lsst/ci_imsim/pull/27 https://github.com/lsst/ci_hsc_gen3/pull/82 https://github.com/lsst/meas_transiNet/pull/9 https://github.com/lsst/pipelines_lsst_io/pull/212
            Hide
            sullivan Ian Sullivan added a comment - - edited
            Show
            sullivan Ian Sullivan added a comment - - edited Jenkins (including ci_hsc and ci_imsim ): https://ci.lsst.codes/blue/organizations/jenkins/stack-os-matrix/detail/stack-os-matrix/37032/pipeline
            Hide
            sullivan Ian Sullivan added a comment -

            DRP is happy with these changes as long as ci_hsc and ci_imsim pass on Jenkins, but please check over all of the AP changes.

            Show
            sullivan Ian Sullivan added a comment - DRP is happy with these changes as long as ci_hsc and ci_imsim pass on Jenkins, but please check over all of the AP changes.
            Hide
            krzys Krzysztof Findeisen added a comment - - edited

            Most of the changes look good, but the metrics handling needs some more work. In particular, there needs to be a PR in the verify_metrics package to accommodate the metrics you've created or renamed, and our runtime graphs in SQuaSH will need to be updated as well – especially if we want to plot the old and new metrics on the same graph!

            Show
            krzys Krzysztof Findeisen added a comment - - edited Most of the changes look good, but the metrics handling needs some more work. In particular, there needs to be a PR in the verify_metrics package to accommodate the metrics you've created or renamed, and our runtime graphs in SQuaSH will need to be updated as well – especially if we want to plot the old and new metrics on the same graph!
            Hide
            sullivan Ian Sullivan added a comment -

            All requested changes on the PRs should now be addressed.

            Show
            sullivan Ian Sullivan added a comment - All requested changes on the PRs should now be addressed.
            Show
            sullivan Ian Sullivan added a comment - Jenkins passes, with ci_hsc and ci_imsim : https://ci.lsst.codes/blue/organizations/jenkins/stack-os-matrix/detail/stack-os-matrix/37035/pipeline/47/
            Hide
            krzys Krzysztof Findeisen added a comment -

            Looks good! I'm looking forward to seeing how the new pipeline performs.

            Show
            krzys Krzysztof Findeisen added a comment - Looks good! I'm looking forward to seeing how the new pipeline performs.

              People

              Assignee:
              sullivan Ian Sullivan
              Reporter:
              sullivan Ian Sullivan
              Reviewers:
              Krzysztof Findeisen
              Watchers:
              Ian Sullivan, Krzysztof Findeisen
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Jenkins

                  No builds found.