Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-35504

Unify ap_pipe configs

    XMLWordPrintable

    Details

    • Type: Story
    • Status: To Do
    • Resolution: Unresolved
    • Fix Version/s: None
    • Component/s: ap_pipe
    • Labels:

      Description

      In digging a bit deeper into DM-35359, John, Ken, and I (re?)discovered that all configs in ap_pipe/pipelines/ApTemplate.yaml (and other "base pipelines" in the same directory) are thrown out by HSC and LsstCamImSim due to DM-34699 and DM-31063.

      What this means is, for the time being, if you want to change a pipeline configuration, you need to either add it to ap_pipe/config/$CAMERA/whateverTask.py or you need to add it to the camera-specific pipeline. This is a temporary state of affairs that nevertheless breaks how "base pipelines" are supposed to work, as a place to set default pipeline configs.

      This ticket is to remedy the situation. I believe this involves the following steps:

      (1) Audit the configs in ap_pipe/config, most notably the ones in ap_pipe/config/$CAMERA/whateverTask.py, to decide which ones should be task-level configs vs. AP pipeline specific configs (vs thrown out entirely)

      (2) Move configs to their appropriate new home

      (3) Remove the entire ap_pipe/config directory (I think?)

      Then configs can only be set in two places: task defaults or pipelines themselves, neither of which overwrites the other. (Or, you know, at runtime on the command line, when living dangerously.)

        Attachments

          Issue Links

            Activity

            Hide
            krzys Krzysztof Findeisen added a comment - - edited

            Note that the ap_pipe/config configs can't be removed until the obs package configs are gone, as that would cause the obs configs to be applied again. I was never able to find a way to suppress the obs configs without also taking out the base pipeline ones.

            As for where things are configured, the motivation for the current arrangement was that it would make it easier to synchronize any changes to the obs package configs (which are still being used as the central config location by the DRP team) when the time came to remove those. If you're planning to merge the ap_pipe/config files into the pipelines, do you have an alternative way to identify "shared" changes that should have been part of AP? I'm worried that if we don't sync anything, whoever is tasked with removing the obs configs will mistake configs that should be task-level for ones that are DRP-specific.

            Show
            krzys Krzysztof Findeisen added a comment - - edited Note that the ap_pipe/config configs can't be removed until the obs package configs are gone, as that would cause the obs configs to be applied again. I was never able to find a way to suppress the obs configs without also taking out the base pipeline ones. As for where things are configured, the motivation for the current arrangement was that it would make it easier to synchronize any changes to the obs package configs (which are still being used as the central config location by the DRP team) when the time came to remove those. If you're planning to merge the ap_pipe/config files into the pipelines, do you have an alternative way to identify "shared" changes that should have been part of AP? I'm worried that if we don't sync anything, whoever is tasked with removing the obs configs will mistake configs that should be task-level for ones that are DRP-specific.
            Hide
            mrawls Meredith Rawls added a comment -

            Great point Krzysztof Findeisen, thank you. Does a ticket exist for moving DRP configs from obs packages into drp_pipe? If I'm fully understanding the situation, it sounds like it would be simplest to set that as a blocker for this ticket. The idea being that only what I would call "true camera-required configs" remain in the obs packages.

            Show
            mrawls Meredith Rawls added a comment - Great point Krzysztof Findeisen , thank you. Does a ticket exist for moving DRP configs from obs packages into drp_pipe? If I'm fully understanding the situation, it sounds like it would be simplest to set that as a blocker for this ticket. The idea being that only what I would call "true camera-required configs" remain in the obs packages.
            Hide
            krzys Krzysztof Findeisen added a comment -

            Thanks, a blocker sounds perfect. I think the ticket in question is DM-31047.

            Show
            krzys Krzysztof Findeisen added a comment - Thanks, a blocker sounds perfect. I think the ticket in question is DM-31047 .

              People

              Assignee:
              Unassigned Unassigned
              Reporter:
              mrawls Meredith Rawls
              Watchers:
              John Parejko, Kenneth Herner, Krzysztof Findeisen, Meredith Rawls
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

                Dates

                Created:
                Updated:

                  Jenkins

                  No builds found.