Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-35504

Unify ap_pipe configs

    XMLWordPrintable

Details

    • Story
    • Status: To Do
    • Resolution: Unresolved
    • None
    • ap_pipe

    Description

      In digging a bit deeper into DM-35359, John, Ken, and I (re?)discovered that all configs in ap_pipe/pipelines/ApTemplate.yaml (and other "base pipelines" in the same directory) are thrown out by HSC and LsstCamImSim due to DM-34699 and DM-31063.

      What this means is, for the time being, if you want to change a pipeline configuration, you need to either add it to ap_pipe/config/$CAMERA/whateverTask.py or you need to add it to the camera-specific pipeline. This is a temporary state of affairs that nevertheless breaks how "base pipelines" are supposed to work, as a place to set default pipeline configs.

      This ticket is to remedy the situation. I believe this involves the following steps:

      (1) Audit the configs in ap_pipe/config, most notably the ones in ap_pipe/config/$CAMERA/whateverTask.py, to decide which ones should be task-level configs vs. AP pipeline specific configs (vs thrown out entirely)

      (2) Move configs to their appropriate new home

      (3) Remove the entire ap_pipe/config directory (I think?)

      Then configs can only be set in two places: task defaults or pipelines themselves, neither of which overwrites the other. (Or, you know, at runtime on the command line, when living dangerously.)

      Attachments

        Issue Links

          Activity

            nlust Nate Lust added a comment -

            You should not need a custom executor to use that functionality, you just implement that method on any PipelineTaskConfig and you can have it do whatever you want with applying configs. That method gets called by the standard executor. Let me catch up on this ticket above to see if it can solve the issue here.

            nlust Nate Lust added a comment - You should not need a custom executor to use that functionality, you just implement that method on any PipelineTaskConfig and you can have it do whatever you want with applying configs. That method gets called by the standard executor. Let me catch up on this ticket above to see if it can solve the issue here.
            krzys Krzysztof Findeisen added a comment - - edited

            Then no, it won't. We're looking for a way to prevent the AP pipelines from calling DRP-biased obs config overrides (pending the cleanup in DM-31047), so something that works at the task level rather than the pipeline level won't solve the problem.

            krzys Krzysztof Findeisen added a comment - - edited Then no, it won't. We're looking for a way to prevent the AP pipelines from calling DRP-biased obs config overrides (pending the cleanup in DM-31047 ), so something that works at the task level rather than the pipeline level won't solve the problem.
            nlust Nate Lust added a comment -

            You are trying to prevent Instrument level overrides from triggering at all right?

            nlust Nate Lust added a comment - You are trying to prevent Instrument level overrides from triggering at all right?

            Yes, at least in specific cases (HSC is the worst offender, DECam and ImSim have some stuff, and LATISS has had proper separation of concerns from the beginning).

            krzys Krzysztof Findeisen added a comment - Yes, at least in specific cases (HSC is the worst offender, DECam and ImSim have some stuff, and LATISS has had proper separation of concerns from the beginning).
            nlust Nate Lust added a comment -

            What would you think if I pushed a change such that you could still do

            instrument: lsst.obs.YYYY

            but also

            instrument:

              name: lsst.obs.YYYY

              exclude:

                - label1

                - label2

             

            I think that would solve your problem, and may be more generally useful for future pipelines as well, though maybe we are moving away from too many configs in obs packages. Though it might be nice to always have the escape hatch.

            nlust Nate Lust added a comment - What would you think if I pushed a change such that you could still do instrument: lsst.obs.YYYY but also instrument:   name: lsst.obs.YYYY   exclude:     - label1     - label2   I think that would solve your problem, and may be more generally useful for future pipelines as well, though maybe we are moving away from too many configs in obs packages. Though it might be nice to always have the escape hatch.

            People

              Unassigned Unassigned
              mrawls Meredith Rawls
              Jim Bosch, John Parejko, Kenneth Herner, Krzysztof Findeisen, Meredith Rawls, Nate Lust
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

                Created:
                Updated:

                Jenkins

                  No builds found.