Details
-
Type:
RFC
-
Status: Adopted
-
Resolution: Unresolved
-
Component/s: DM
-
Labels:None
Description
Nate Lust and I have put together a proposal for reorganizing both our top-level packages and moving our Gen3 Pipeline files into new packages. It's a bit too large for Jira ticket description box, so we've put it on a confluence page:
https://confluence.lsstcorp.org/display/~jbosch/Reorganizing+Top-Level+Packages+and+Pipelines
There are a few goals intertwined here, but the highest priority is creating a "good" place to put our Pipeline files and the BPS configuration files we use to run them. This is an ambitious, far-reaching proposal with (currently) a short deadline, because we'd like to move quickly to address at least said location problem, and if other aspects of the proposal appear more controversial in ways that don't affect that primary goal, we'd rather reduce the scope (punting the rest to a new RFC) than extend the deadline. But we're starting with the big proposal because many of these changes are related.
Please use Confluence in-line comments (select text, then hit the button that pops up next to it) for feedback on details, and bottom-of-the-page Confluence comments for big-picture things (which thread a bit better than Jira comments). We can use Jira comments for discussions of the scope and announcements of changes to the proposal that is currently on the table (which will also be reflected in the Confluence text).
Attachments
Issue Links
- is SF-depended by
-
DM-29981 Migrate cp_pipe pipelines as defined in RFC-775
- Done
- is triggering
-
DM-30893 Add faro- and analysis_drp-instrumented pipelines to drp_pipe
- To Do
-
DM-30894 Move ForcedPhot PipelineTasks to pipe_tasks
- To Do
-
DM-30896 Move "multiband" coadd-processing tasks to drp_tasks
- To Do
-
DM-30897 Remove pipe_tasks dependency on ip_isr
- To Do
-
DM-30899 Stop using obs_* configurations in jointcal
- To Do
-
DM-30901 Trim lsst_distrib dependencies to maximize coverage by *_pipe packages
- To Do
-
DM-31047 Remove configuration from obs_* packages after Gen2 removal
- To Do
-
DM-33453 Update ap_pipe pipelines to match drp_pipe naming
- In Review
-
DM-29823 Experiment with the task_base concept
- Done
-
DM-30891 Create drp_pipe and move all DRP pipeline definitions there
- Done
-
DM-30892 Add BPS config-file templates to drp_pipe
- Done
-
DM-30895 Move coaddition tasks to drp_tasks
- Done
-
DM-30900 Stop using obs_* configurations in fgcmcal
- Done
-
DM-31063 Copy configs from obs_* packages to ap_pipe
- Done
-
DM-35579 Remove pipeline redirects in pipe_tasks and obs_ packages
- Done
-
RFC-959 Reorganization of ap_pipe and ap_verify directory structures without backwards compatibility
- Proposed
- relates to
-
DM-28862 Make Gen3 transform Source Table tasks work on all cameras
- Done
-
DM-39212 Move ingredient pipeline definitions in cp_pipe and cp_verify to the pipelines directory
- Done
-
DM-39214 Move ingredient pipeline definitions in ap_pipe and ap_verify to the pipelines directory
- Done
-
DM-40210 Clean up ap_pipe and ap_verify pipelines
- Done
-
DM-30668 Create drp_pipe and migrate DRP.yaml and related pipelines
- Invalid
-
DM-36796 Rename config/pipeline directories in ap_pipe and ap_verify
- Invalid
-
RFC-783 Replace Task metadata class with pure Python Mapping
- Implemented
-
RFC-927 Clarify pipeline definition YAML directory structures
- Implemented
-
DM-39796 Update task config defaults to LSST values
- Done
-
DM-17321 Provide mechanism for storing configuration for specific processing campaigns
- Won't Fix
- mentioned in
-
Page Loading...
-
Page Loading...
-
Page Loading...
-
Page Loading...
-
Page Loading...
-
Page Loading...
-
Page Loading...
-
Page Loading...
-
Page Loading...
-
Page Loading...
-
Page Loading...
-
Page Loading...
-
Page Loading...
There is not much that needs to be done on the AP side to implement this RFC. Some recent work that incorporates the design decisions are
DM-29338andDM-29221, though I don't think it's necessary to label them as "triggered by" the RFC.