Uploaded image for project: 'Request For Comments'
  1. Request For Comments
  2. RFC-775

Reorganize pipelines and packages at the top of the Science Pipelines codebase

    XMLWordPrintable

    Details

    • Type: RFC
    • Status: Adopted
    • Resolution: Unresolved
    • Component/s: DM
    • Labels:
      None

      Description

      Nate Lust and I have put together a proposal for reorganizing both our top-level packages and moving our Gen3 Pipeline files into new packages. It's a bit too large for Jira ticket description box, so we've put it on a confluence page:

      https://confluence.lsstcorp.org/display/~jbosch/Reorganizing+Top-Level+Packages+and+Pipelines

      There are a few goals intertwined here, but the highest priority is creating a "good" place to put our Pipeline files and the BPS configuration files we use to run them. This is an ambitious, far-reaching proposal with (currently) a short deadline, because we'd like to move quickly to address at least said location problem, and if other aspects of the proposal appear more controversial in ways that don't affect that primary goal, we'd rather reduce the scope (punting the rest to a new RFC) than extend the deadline. But we're starting with the big proposal because many of these changes are related.

      Please use Confluence in-line comments (select text, then hit the button that pops up next to it) for feedback on details, and bottom-of-the-page Confluence comments for big-picture things (which thread a bit better than Jira comments). We can use Jira comments for discussions of the scope and announcements of changes to the proposal that is currently on the table (which will also be reflected in the Confluence text).

        Attachments

          Issue Links

            Activity

            Hide
            sullivan Ian Sullivan added a comment -

            There is not much that needs to be done on the AP side to implement this RFC. Some recent work that incorporates the design decisions are DM-29338 and DM-29221, though I don't think it's necessary to label them as "triggered by" the RFC.

            Show
            sullivan Ian Sullivan added a comment - There is not much that needs to be done on the AP side to implement this RFC. Some recent work that incorporates the design decisions are DM-29338 and DM-29221 , though I don't think it's necessary to label them as "triggered by" the RFC.
            Hide
            krzys Krzysztof Findeisen added a comment - - edited

            I think we still need to move some task-level config overrides out of the obs packages for AP, though presumably this move should be done simultaneously for all the pipeline packages. Is there a ticket for this? The closest that I can see are DM-30899 and DM-30900, which are specific to jointcal and fgcmcal, respectively. Or will the configs stay behind for the sake of Gen 2?

            Removing the configs from the obs packages would likely break ap_verify and its data sets, though DM-26140 can address this. I propose it be classified as a triggering ticket.

            We may need to remove the test dependencies of ap_pipe and ap_verify on obs_lsst, though I'll need to look into exactly what we are getting from the obs package. In ap_verify, there is at least a dependency on the CameraMapper (in Gen 2) or Instrument (in Gen 3); in ap_pipe it's less clear whether this is "data access", "configuration", or both.

            Show
            krzys Krzysztof Findeisen added a comment - - edited I think we still need to move some task-level config overrides out of the obs packages for AP, though presumably this move should be done simultaneously for all the pipeline packages. Is there a ticket for this? The closest that I can see are DM-30899 and DM-30900 , which are specific to jointcal and fgcmcal , respectively. Or will the configs stay behind for the sake of Gen 2? Removing the configs from the obs packages would likely break ap_verify and its data sets, though DM-26140 can address this. I propose it be classified as a triggering ticket. We may need to remove the test dependencies of ap_pipe and ap_verify on obs_lsst , though I'll need to look into exactly what we are getting from the obs package. In ap_verify , there is at least a dependency on the CameraMapper (in Gen 2) or Instrument (in Gen 3); in ap_pipe it's less clear whether this is "data access", "configuration", or both.
            Hide
            jbosch Jim Bosch added a comment -

            I was thinking that most configs would stay behind for Gen2, until that is gone, but I did plan to also write a ticket or two for that post-Gen2-removal migration, and forgot.  I'll write a vague one to capture the fact that it's a goal of this RFC, as I think we'll learn a lot more about the details from earlier tickets, especially about what's really common to AP and DRP.

            If AP can move AP-only configs out of obs* packages earlier on DM-26140, because your primary Gen2 execution harness (ap_verify) already goes through ap_pipe, I'm all for doing that sooner and calling it a triggering ticket.  I think DRP is stuck with our configs being mostly in obs_* packages for a while longer because it isn't worth the effort to make our Gen2 execution harness (pipe_drivers) look somewhere new.

            Show
            jbosch Jim Bosch added a comment - I was thinking that most configs would stay behind for Gen2, until that is gone, but I did plan to also write a ticket or two for that post-Gen2-removal migration, and forgot.  I'll write a vague one to capture the fact that it's a goal of this RFC, as I think we'll learn a lot more about the details from earlier tickets, especially about what's really common to AP and DRP. If AP can move AP-only configs out of obs* packages earlier on DM-26140 , because your primary Gen2 execution harness (ap_verify) already goes through ap_pipe, I'm all for doing that sooner and calling it a triggering ticket.  I think DRP is stuck with our configs being mostly in obs_* packages for a while longer because it isn't worth the effort to make our Gen2 execution harness (pipe_drivers) look somewhere new.
            Hide
            jbosch Jim Bosch added a comment -

            Some of the "let's experiment for the details of this design" promised on this RFC has now happened on DM-30891, and that's revealed that the idea of committing expanded pipelines to the (e.g.) drp_pipe repo just isn't sound. See https://community.lsst.org/t/expanding-and-documenting-pipeline-definitions/6118 for follow-up discussion.

            Show
            jbosch Jim Bosch added a comment - Some of the "let's experiment for the details of this design" promised on this RFC has now happened on DM-30891 , and that's revealed that the idea of committing expanded pipelines to the (e.g.) drp_pipe repo just isn't sound. See https://community.lsst.org/t/expanding-and-documenting-pipeline-definitions/6118 for follow-up discussion.
            Hide
            lskelvin Lee Kelvin added a comment -

            The pipeline organizational changes that were initiated as part of this RFC have now been fully implemented in drp_pipe, ap_pipe, ap_verify, cp_pipe and cp_verify. For further discussion and clarification, see RFC-927.

            Show
            lskelvin Lee Kelvin added a comment - The pipeline organizational changes that were initiated as part of this RFC have now been fully implemented in drp_pipe , ap_pipe , ap_verify , cp_pipe and cp_verify . For further discussion and clarification, see RFC-927 .

              People

              Assignee:
              jbosch Jim Bosch
              Reporter:
              jbosch Jim Bosch
              Watchers:
              Christopher Waters, Dan Taranu, Eric Bellm, Gregory Dubois-Felsmann, Hsin-Fang Chiang, Ian Sullivan, Jim Bosch, John Parejko, Krzysztof Findeisen, Lee Kelvin, Nate Lust, Simon Krughoff, Tatiana Goldina, Tim Jenness
              Votes:
              0 Vote for this issue
              Watchers:
              14 Start watching this issue

                Dates

                Created:
                Updated:
                Planned End:

                  Jenkins

                  No builds found.