Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-27385

Parallelize gen3 PTC code



    • Improvement
    • Status: To Do
    • Resolution: Unresolved
    • None
    • cp_pipe
    • None


      The PTC code needs to operate on paired exposures, but these can not be calculated with the current gen3 middleware.  The proposed solution is to add an initial pairing task, that will read the full set of ISR processed exposures (for a given detector), construct pairs, and then write a `ptcPairMetadata` dictionary for each of the input exposures.  These dictionaries will have butler dimensions of (instrument, exposure, detector), and can then be used as the primary input to the `extract` task.  The contents of the `ptcPairMetadata` will contain the expId pairs, as well as a boolean indicating if the extract task should run (in the case of the first expId in a pair) or populate an empty/dummy output (for the second expId in the pair). 

      The `PtcExtractTask` can then be updated to read the input metadata, as well as a deferred load of the full set of input exposures.  The contents of the metadata can be used to fully load the two exposures of interest, run the PTC extract, and write out the partial PTC datasets.  The `PtcSolveTask` can then run as normal on those inputs.

      A "job number" flow chart for N input exposures is then:

      1. N `IsrTask` calls to pre-process the raw exposures.
      2. 1 `PtcPairTask` call to determine the set of pairs and write the N `ptcPairMetadata` files. This can likely also used deferred loads to prevent memory issues.
      3. N/2 active `PtcExtractTask` calls to generate partial PTC datasets, and N/2 inactive `PtcExtractTask` calls to write empty data for the second exposures in each pair.
      4. 1 `PtcSolveTask` call to collect all partial PTC datasets (and empty datasets that are ignored) into a final PTC dataset.


        Issue Links


            There are no comments yet on this issue.


              plazas Andrés Alejandro Plazas Malagón
              czw Christopher Waters
              Andrés Alejandro Plazas Malagón, Christopher Waters
              0 Vote for this issue
              2 Start watching this issue




                  No builds found.