Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-22163

Add config writing to PipelineTask execution logic

    Details

    • Story Points:
      3
    • Team:
      Data Release Production

      Description

      PipelineTask execution should automatically write configs to the data repository.

      There are some details about how to do this are still up in the air:

      • Should these appear in the QuantumGraph, so they can be consumed as Inputs or InitInputs by downstream tasks?  That would be one way to unblock part of DM-21885, but it is not the only way.
      • Should these be per-Run?  (Not if we want to support config-heterogeneous QuantumGraphs for processing data from multiple instruments together, as Nate Lust and I have recently discussed.)
      • Should these be per-Quantum?  (If so, that's a lot of duplication in the typical case.)

      DM-21849 may provide good way to make them neither per-Run nor per-Quantum: the execution system will know exactly how many distinct configs it needs for a run, and making them "nonsingular" datasets as described there would let us write them without duplication while associating the same config dataset with multiple quanta.

        Attachments

          Issue Links

            Activity

            There are no comments yet on this issue.

              People

              • Assignee:
                Unassigned
                Reporter:
                jbosch Jim Bosch
                Watchers:
                Andy Salnikov, Christopher Waters, Jim Bosch, Krzysztof Findeisen, Nate Lust
              • Votes:
                0 Vote for this issue
                Watchers:
                5 Start watching this issue

                Dates

                • Created:
                  Updated:

                  Summary Panel