Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-26140

Centralize Gen 3 pipeline configuration info for ap_verify datasets

    XMLWordPrintable

    Details

    • Type: Improvement
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: ap_verify
    • Labels:
      None
    • Story Points:
      12
    • Sprint:
      AP F21-3 (August), AP F21-4 (September), AP F21-5 (October)
    • Team:
      Alert Production

      Description

      As implemented in DM-21919, ap_verify requires a lot of in-code hacks to properly configure a dataset in Gen 3 (including specification of an instrument). Find a way to delegate as much of this as possible to something in each dataset that acts like a "pipeline config".

      The description is deliberately vague because this kind of delegation is harder than it sounds; see discussion in #dm-middleware.

        Attachments

          Issue Links

            Activity

            Hide
            mrawls Meredith Rawls added a comment -

            Thanks for tackling this, Krzysztof, and apologies for the delay. I think this ticket realizes the vague idea for pipeline organization I had in my head, but I didn't quite have enough yaml subset pipeline-writing experience to implement from the get-go.

            One question that will need a followup ticket is when we should use the new lsst.ip.diffim.getTemplate.GetMultiTractCoaddTemplateTask + lsst.pipe.tasks.imageDifference.ImageDifferenceFromTemplateTask instead of just lsst.pipe.tasks.imageDifference.ImageDifferenceTask.

            That workflow is necessary for multi-tract differencing, has its own config + DM-30210 complications, etc.

            The only things I'm confused about are exactly how the camera-specific ApVerify pipelines with fakes are supposed to work (see GitHub comments on the ap_verify PR).

            I defer to Nate for any suggestions on making framework things better, but I'm very pleased with the ApPipe pipeline standardization, and as far as I can see, both it and the ApVerify pipelines do what they should without too many complicated import trains.

            Show
            mrawls Meredith Rawls added a comment - Thanks for tackling this, Krzysztof, and apologies for the delay. I think this ticket realizes the vague idea for pipeline organization I had in my head, but I didn't quite have enough yaml subset pipeline-writing experience to implement from the get-go. One question that will need a followup ticket is when we should use the new lsst.ip.diffim.getTemplate.GetMultiTractCoaddTemplateTask + lsst.pipe.tasks.imageDifference.ImageDifferenceFromTemplateTask instead of just lsst.pipe.tasks.imageDifference.ImageDifferenceTask. That workflow is necessary for multi-tract differencing, has its own config + DM-30210 complications, etc. The only things I'm confused about are exactly how the camera-specific ApVerify pipelines with fakes are supposed to work (see GitHub comments on the ap_verify PR). I defer to Nate for any suggestions on making framework things better, but I'm very pleased with the ApPipe pipeline standardization, and as far as I can see, both it and the ApVerify pipelines do what they should without too many complicated import trains.
            Hide
            nlust Nate Lust added a comment -

            I had a few minor comments, but it mostly looks good as far as structure goes, and gives me so good feedback and ideas surrounding RFC-775 pipeline building. I am taking myself off the reviewers list.

            Show
            nlust Nate Lust added a comment - I had a few minor comments, but it mostly looks good as far as structure goes, and gives me so good feedback and ideas surrounding RFC-775 pipeline building. I am taking myself off the reviewers list.
            Hide
            mrawls Meredith Rawls added a comment -

            If you ping me when you're done making changes, I can take a final look. I think I like Nate's suggestion to use "parameters" to streamline e.g. coadd/fakes names, both pre- and post- DM-30210.

            Show
            mrawls Meredith Rawls added a comment - If you ping me when you're done making changes, I can take a final look. I think I like Nate's suggestion to use "parameters" to streamline e.g. coadd/fakes names, both pre- and post- DM-30210 .
            Hide
            krzys Krzysztof Findeisen added a comment -

            Meredith Rawls, can you take a look at the new and improved pipelines? Thanks!

            Show
            krzys Krzysztof Findeisen added a comment - Meredith Rawls , can you take a look at the new and improved pipelines? Thanks!
            Hide
            mrawls Meredith Rawls added a comment -

            This looks great - thank you for standardizing all the indents and greatly improving all the configs! There's a lot here but as far as I can tell it all looks self-consistent. Let's get it merged, thank you for your attention to detail.

            Show
            mrawls Meredith Rawls added a comment - This looks great - thank you for standardizing all the indents and greatly improving all the configs! There's a lot here but as far as I can tell it all looks self-consistent. Let's get it merged, thank you for your attention to detail.

              People

              Assignee:
              krzys Krzysztof Findeisen
              Reporter:
              krzys Krzysztof Findeisen
              Reviewers:
              Meredith Rawls
              Watchers:
              Krzysztof Findeisen, Lee Kelvin, Meredith Rawls, Nate Lust
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Jenkins

                  No builds found.