Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-22163

Add config writing to PipelineTask execution logic

    XMLWordPrintable

    Details

    • Story Points:
      3
    • Team:
      Data Release Production

      Description

      PipelineTask execution should automatically write configs to the data repository.

      There are some details about how to do this are still up in the air:

      • Should these appear in the QuantumGraph, so they can be consumed as Inputs or InitInputs by downstream tasks?  That would be one way to unblock part of DM-21885, but it is not the only way.
      • Should these be per-Run?  (Not if we want to support config-heterogeneous QuantumGraphs for processing data from multiple instruments together, as Nate Lust and I have recently discussed.)
      • Should these be per-Quantum?  (If so, that's a lot of duplication in the typical case.)

      DM-21849 may provide good way to make them neither per-Run nor per-Quantum: the execution system will know exactly how many distinct configs it needs for a run, and making them "nonsingular" datasets as described there would let us write them without duplication while associating the same config dataset with multiple quanta.

        Attachments

          Issue Links

            Activity

            Hide
            tjenness Tim Jenness added a comment -

            I'm certain we are writing configs to butler during execution so I think this ticket got lost somehow.

            Show
            tjenness Tim Jenness added a comment - I'm certain we are writing configs to butler during execution so I think this ticket got lost somehow.

              People

              Assignee:
              Unassigned Unassigned
              Reporter:
              jbosch Jim Bosch
              Watchers:
              Andy Salnikov, Christopher Waters, Jim Bosch, Krzysztof Findeisen, Nate Lust, Tim Jenness
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

                Dates

                Created:
                Updated:

                  Jenkins

                  No builds found.