Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-27385

Parallelize gen3 PTC code

    XMLWordPrintable

    Details

    • Type: Improvement
    • Status: To Do
    • Resolution: Unresolved
    • Fix Version/s: None
    • Component/s: cp_pipe
    • Labels:
      None

      Description

      The PTC code needs to operate on paired exposures, but these can not be calculated with the current gen3 middleware.  The proposed solution is to add an initial pairing task, that will read the full set of ISR processed exposures (for a given detector), construct pairs, and then write a `ptcPairMetadata` dictionary for each of the input exposures.  These dictionaries will have butler dimensions of (instrument, exposure, detector), and can then be used as the primary input to the `extract` task.  The contents of the `ptcPairMetadata` will contain the expId pairs, as well as a boolean indicating if the extract task should run (in the case of the first expId in a pair) or populate an empty/dummy output (for the second expId in the pair). 

      The `PtcExtractTask` can then be updated to read the input metadata, as well as a deferred load of the full set of input exposures.  The contents of the metadata can be used to fully load the two exposures of interest, run the PTC extract, and write out the partial PTC datasets.  The `PtcSolveTask` can then run as normal on those inputs.

      A "job number" flow chart for N input exposures is then:

      1. N `IsrTask` calls to pre-process the raw exposures.
      2. 1 `PtcPairTask` call to determine the set of pairs and write the N `ptcPairMetadata` files. This can likely also used deferred loads to prevent memory issues.
      3. N/2 active `PtcExtractTask` calls to generate partial PTC datasets, and N/2 inactive `PtcExtractTask` calls to write empty data for the second exposures in each pair.
      4. 1 `PtcSolveTask` call to collect all partial PTC datasets (and empty datasets that are ignored) into a final PTC dataset.

        Attachments

          Issue Links

            Activity

            There are no comments yet on this issue.

              People

              Assignee:
              plazas Andrés Alejandro Plazas Malagón
              Reporter:
              czw Christopher Waters
              Watchers:
              Andrés Alejandro Plazas Malagón, Christopher Waters
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

                Dates

                Created:
                Updated:

                  CI Builds

                  No builds found.