Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-32988

Update all ApPipe pipelines to use multi-tract difference imaging

    XMLWordPrintable

Details

    • Story
    • Status: Done
    • Resolution: Done
    • None
    • ap_pipe
    • None

    Description

      Following the success of DM-32245, it is pretty apparent that all the various flavors of ApPipe.yaml should use multi-tract difference imaging by default. At present, ApPipe.yaml is single-tract and ApPipeMultiTractFakes.yaml is multi-tract (with fakes, as the name suggests).

      This ticket is to standardize the situation further:

      • ApPipe should use multi-tract difference imaging
      • Rename ApPipeMultiTractFakes (and its camera-specific derivatives) to just ApPipeFakes
      • Remove lsst-imSim's ApPipeFakes because it's already all fake
      • Use yaml parameters throughout for setting e.g. fakesType and coaddName
      • Ensure all pipeline imports (especially in camera-specific pipelines) are as straightforward as possible
      • Run each camera-specific pipeline to make sure nothing broke

      Attachments

        Issue Links

          Activity

            krzys Krzysztof Findeisen added a comment - - edited

            I'm listing this as a blocker for DM-28628, which will probably involve putting both new calibs and new templates into the dataset. The templates generated by DM-29857 use the finer-grained hsc_rings_v1 skymap, and therefore cannot be processed without multi-tract support.

            (We can update just the calibs, if need be, but I think having everything up-to-date would be better.)

            krzys Krzysztof Findeisen added a comment - - edited I'm listing this as a blocker for DM-28628 , which will probably involve putting both new calibs and new templates into the dataset. The templates generated by DM-29857 use the finer-grained hsc_rings_v1 skymap, and therefore cannot be processed without multi-tract support. (We can update just the calibs, if need be, but I think having everything up-to-date would be better.)
            mrawls Meredith Rawls added a comment - - edited

            This work is being completed on PREOPS-931.

            LOL no it's not, we're going to move it here, stay tuned.

            mrawls Meredith Rawls added a comment - - edited This work is being completed on PREOPS-931. LOL no it's not, we're going to move it here, stay tuned.

            Current status.

            • Ken updated ap_pipe/pipelines/ApPipe.yaml to use ImageDifferenceFromTemplateTask (multi-tract differencing) plus the necessary GetMultiTractCoaddTemplateTask prerequisite, in lieu of single-tract ImageDifferenceTask
            • Meredith updated the ap_pipe fakes-related pipelines to be consistent with this, as the ticket description suggests
            • Krzysztof helped us realize the three ap_verify datasets needed to have updated pipelines, since they all have their own custom pipelines that import from ap_pipe/pipelines, so now there are three more PRs
            • TO DO (Ken): test runs of (1) ap_verify.py on both CI datasets and (2) ApPipe.yaml on a small HSC dataset to make sure nothing horrible happened
            • Assuming nothing is broken, the work to tidy up various pipelines in ap_verify to use the new ap_pipe fakes pipelines and tidy up the overall ApVerifyWithFakes situation (and/or the messiness of per-dataset pipelines in general!) is punted to DM-32694

            Let me know if I've missed anything or misrepresented anything!

            mrawls Meredith Rawls added a comment - Current status. Ken updated ap_pipe/pipelines/ApPipe.yaml to use ImageDifferenceFromTemplateTask (multi-tract differencing) plus the necessary GetMultiTractCoaddTemplateTask prerequisite, in lieu of single-tract ImageDifferenceTask Meredith updated the ap_pipe fakes-related pipelines to be consistent with this, as the ticket description suggests Krzysztof helped us realize the three ap_verify  datasets  needed to have updated pipelines, since they all have their own custom pipelines that import from ap_pipe/pipelines, so now there are three more PRs TO DO (Ken): test runs of (1) ap_verify.py on both CI datasets and (2) ApPipe.yaml on a small HSC dataset to make sure nothing horrible happened Assuming nothing is broken, the work to tidy up various pipelines in ap_verify to use the new ap_pipe fakes pipelines and tidy up the overall ApVerifyWithFakes situation (and/or the messiness of per-dataset pipelines in general!) is punted to DM-32694 Let me know if I've missed anything or misrepresented anything!
            mrawls Meredith Rawls added a comment - Links to all PRs, because Jira is at least as confused as I am 0. https://github.com/lsst/ap_pipe/pull/99   1. https://github.com/lsst/ap_verify_hits2015/pull/37 2. https://github.com/lsst/ap_verify_ci_hits2015/pull/32   3. https://github.com/lsst/ap_verify_ci_cosmos_pdr2/pull/20  

            Everything looks good after the new changes.

            krzys Krzysztof Findeisen added a comment - Everything looks good after the new changes.

            ap_pipe and the three ap_verify_<dataset> packages have all been merged now.

            kherner Kenneth Herner added a comment - ap_pipe and the three ap_verify_<dataset> packages have all been merged now.

            People

              kherner Kenneth Herner
              mrawls Meredith Rawls
              Krzysztof Findeisen
              Ian Sullivan, Kenneth Herner, Krzysztof Findeisen, Meredith Rawls
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Jenkins

                  No builds found.