Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-39858

Integrate new CalibrateImageTask with AP pipeline

    XMLWordPrintable

Details

    Description

      Once the new CalibrateImageTask is merged, we can try to integrate it into the AP pipeline. As a first step, add a new ap_verify pipeline definition that includes CalibrateImageTask, so that we can trivially compare with pipelines using the existing characterizeImage/calibrate tasks.

      Attachments

        Issue Links

          Activity

            Parejkoj John Parejko added a comment - - edited

            krzys: would you be willing to review the new pipelines here? I believe I've updated them to match the changes you made on DM-40210, but it's clumsy to compare them with the current pipelines, so I'm very open for ideas on how to make this more manageable. Would it be better to put all the possibilities in the same ingredients files, and then have separate `XCalibrateImage.yaml` for the instrument-specific ones? Suggestions, please!

            I've only done the imSim files for now.

            PRs:

            New PRs:

            Parejkoj John Parejko added a comment - - edited krzys : would you be willing to review the new pipelines here? I believe I've updated them to match the changes you made on DM-40210 , but it's clumsy to compare them with the current pipelines, so I'm very open for ideas on how to make this more manageable. Would it be better to put all the possibilities in the same ingredients files, and then have separate `XCalibrateImage.yaml` for the instrument-specific ones? Suggestions, please! I've only done the imSim files for now. PRs: https://github.com/lsst/ap_pipe/pull/147 https://github.com/lsst/ap_verify/pull/194 https://github.com/lsst/verify_metrics/pull/39 https://github.com/lsst/verify/pull/116 https://github.com/lsst/analysis_ap/pull/33 New PRs: https://github.com/lsst/obs_lsst/pull/465 https://github.com/lsst/ap_verify_ci_hits2015/pull/49 https://github.com/lsst/ap_verify_ci_dc2/pull/5 https://github.com/lsst/ap_verify_ci_cosmos_pdr2/pull/35
            krzys Krzysztof Findeisen added a comment - - edited

            I recommend adding the pipelines to ap_verify datasets (just DC2?) and CI as part of this issue; the config file for the latter is https://github.com/lsst-dm/jenkins-dm-jobs/blob/main/etc/scipipe/ap_verify.yaml.

            krzys Krzysztof Findeisen added a comment - - edited I recommend adding the pipelines to ap_verify datasets (just DC2?) and CI as part of this issue; the config file for the latter is https://github.com/lsst-dm/jenkins-dm-jobs/blob/main/etc/scipipe/ap_verify.yaml .
            Parejkoj John Parejko added a comment - - edited

            I've updated all the PRs based on your comments and our conversation, and added 4 new PRs (linked above) for the changes to the ci datasets and obs_lsst config. Please take another look.

            HSC is failing in ap_verify right now, due to one of the chips failing astrometry. I haven't been able to figure out why: the image has reasonable sources detected on it. I think sorting that out will be part of the subsequent "validate new task" ticket.

            I merged the analysis_ap and verify tickets, as they were independent of the other changes, and it'll make it easier for me to use them in testing.

            I'll add the new pipelines to Jenkins CI and Chronagraph on DM-40397.

            Parejkoj John Parejko added a comment - - edited I've updated all the PRs based on your comments and our conversation, and added 4 new PRs (linked above) for the changes to the ci datasets and obs_lsst config. Please take another look. HSC is failing in ap_verify right now, due to one of the chips failing astrometry. I haven't been able to figure out why: the image has reasonable sources detected on it. I think sorting that out will be part of the subsequent "validate new task" ticket. I merged the analysis_ap and verify tickets, as they were independent of the other changes, and it'll make it easier for me to use them in testing. I'll add the new pipelines to Jenkins CI and Chronagraph on DM-40397 .
            Parejkoj John Parejko added a comment - Jenkins: https://rubin-ci.slac.stanford.edu/blue/organizations/jenkins/stack-os-matrix/detail/stack-os-matrix/43/pipeline

            Thanks, krzys for all the comments: it's definitely better organized now. Now to figure out why HSC is failing and do some other validation.

            Parejkoj John Parejko added a comment - Thanks, krzys for all the comments: it's definitely better organized now. Now to figure out why HSC is failing and do some other validation.

            People

              Parejkoj John Parejko
              Parejkoj John Parejko
              Krzysztof Findeisen
              Eric Bellm, Ian Sullivan, John Parejko, Krzysztof Findeisen, Meredith Rawls
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Jenkins

                  No builds found.