Details
-
Type:
RFC
-
Status: Adopted
-
Resolution: Unresolved
-
Component/s: DM
-
Labels:None
Description
Nate Lust and I have put together a proposal for reorganizing both our top-level packages and moving our Gen3 Pipeline files into new packages. It's a bit too large for Jira ticket description box, so we've put it on a confluence page:
https://confluence.lsstcorp.org/display/~jbosch/Reorganizing+Top-Level+Packages+and+Pipelines
There are a few goals intertwined here, but the highest priority is creating a "good" place to put our Pipeline files and the BPS configuration files we use to run them. This is an ambitious, far-reaching proposal with (currently) a short deadline, because we'd like to move quickly to address at least said location problem, and if other aspects of the proposal appear more controversial in ways that don't affect that primary goal, we'd rather reduce the scope (punting the rest to a new RFC) than extend the deadline. But we're starting with the big proposal because many of these changes are related.
Please use Confluence in-line comments (select text, then hit the button that pops up next to it) for feedback on details, and bottom-of-the-page Confluence comments for big-picture things (which thread a bit better than Jira comments). We can use Jira comments for discussions of the scope and announcements of changes to the proposal that is currently on the table (which will also be reflected in the Confluence text).
Attachments
Issue Links
- is SF-depended by
-
DM-29981 Migrate cp_pipe pipelines as defined in RFC-775
- Done
- is triggering
-
- To Do
-
- To Do
-
- To Do
-
- To Do
-
- To Do
-
- To Do
-
- To Do
-
- To Do
-
- In Progress
-
- Done
-
- Done
-
- Done
-
- Done
-
- Done
- relates to
-
- To Do
-
- Done
-
- Invalid
-
- Implemented
-
- Won't Fix
- mentioned in
-
Page Loading...
-
-
-
-
-
-
-
-
-
-
-
-
Activity
I think we still need to move some task-level config overrides out of the obs packages for AP, though presumably this move should be done simultaneously for all the pipeline packages. Is there a ticket for this? The closest that I can see are DM-30899 and DM-30900, which are specific to jointcal and fgcmcal, respectively. Or will the configs stay behind for the sake of Gen 2?
Removing the configs from the obs packages would likely break ap_verify and its data sets, though DM-26140 can address this. I propose it be classified as a triggering ticket.
We may need to remove the test dependencies of ap_pipe and ap_verify on obs_lsst, though I'll need to look into exactly what we are getting from the obs package. In ap_verify, there is at least a dependency on the CameraMapper (in Gen 2) or Instrument (in Gen 3); in ap_pipe it's less clear whether this is "data access", "configuration", or both.
Krzysztof Findeisen
added a comment - - edited I think we still need to move some task-level config overrides out of the obs packages for AP, though presumably this move should be done simultaneously for all the pipeline packages. Is there a ticket for this? The closest that I can see are DM-30899 and DM-30900 , which are specific to jointcal and fgcmcal , respectively. Or will the configs stay behind for the sake of Gen 2?
Removing the configs from the obs packages would likely break ap_verify and its data sets, though DM-26140 can address this. I propose it be classified as a triggering ticket.
We may need to remove the test dependencies of ap_pipe and ap_verify on obs_lsst , though I'll need to look into exactly what we are getting from the obs package. In ap_verify , there is at least a dependency on the CameraMapper (in Gen 2) or Instrument (in Gen 3); in ap_pipe it's less clear whether this is "data access", "configuration", or both. I was thinking that most configs would stay behind for Gen2, until that is gone, but I did plan to also write a ticket or two for that post-Gen2-removal migration, and forgot. I'll write a vague one to capture the fact that it's a goal of this RFC, as I think we'll learn a lot more about the details from earlier tickets, especially about what's really common to AP and DRP.
If AP can move AP-only configs out of obs* packages earlier on DM-26140, because your primary Gen2 execution harness (ap_verify) already goes through ap_pipe, I'm all for doing that sooner and calling it a triggering ticket. I think DRP is stuck with our configs being mostly in obs_* packages for a while longer because it isn't worth the effort to make our Gen2 execution harness (pipe_drivers) look somewhere new.
Some of the "let's experiment for the details of this design" promised on this RFC has now happened on DM-30891, and that's revealed that the idea of committing expanded pipelines to the (e.g.) drp_pipe repo just isn't sound. See https://community.lsst.org/t/expanding-and-documenting-pipeline-definitions/6118 for follow-up discussion.
I think
DM-29981(which is already linked) is the current CPP response to this.