# Parallelize gen3 PTC code

XMLWordPrintable

#### Details

• Type: Improvement
• Status: To Do
• Resolution: Unresolved
• Fix Version/s: None
• Component/s:
• Labels:
None
• Team:
Data Release Production

#### Description

The PTC code needs to operate on paired exposures, but these can not be calculated with the current gen3 middleware.  The proposed solution is to add an initial pairing task, that will read the full set of ISR processed exposures (for a given detector), construct pairs, and then write a ptcPairMetadata dictionary for each of the input exposures.  These dictionaries will have butler dimensions of (instrument, exposure, detector), and can then be used as the primary input to the extract task.  The contents of the ptcPairMetadata will contain the expId pairs, as well as a boolean indicating if the extract task should run (in the case of the first expId in a pair) or populate an empty/dummy output (for the second expId in the pair).

The PtcExtractTask can then be updated to read the input metadata, as well as a deferred load of the full set of input exposures.  The contents of the metadata can be used to fully load the two exposures of interest, run the PTC extract, and write out the partial PTC datasets.  The PtcSolveTask can then run as normal on those inputs.

A "job number" flow chart for N input exposures is then:

1. N IsrTask calls to pre-process the raw exposures.
2. 1 PtcPairTask call to determine the set of pairs and write the N ptcPairMetadata files. This can likely also used deferred loads to prevent memory issues.
3. N/2 active PtcExtractTask calls to generate partial PTC datasets, and N/2 inactive PtcExtractTask calls to write empty data for the second exposures in each pair.
4. 1 PtcSolveTask call to collect all partial PTC datasets (and empty datasets that are ignored) into a final PTC dataset.

#### Activity

There are no comments yet on this issue.

#### People

Assignee:
Andrés Alejandro Plazas Malagón
Reporter:
Christopher Waters
Watchers:
Andrés Alejandro Plazas Malagón, Christopher Waters