# ap_verify_hits2015 detectors 2 and 61 broken in gen3

XMLWordPrintable

#### Details

• Type: Story
• Status: To Do
• Resolution: Unresolved
• Fix Version/s: None
• Component/s:
• Labels:
• Team:
• Urgent?:
No

#### Description

Trying to process ap_verify_hits2015 with ap_verify using gen3 without specifying --data-query "detector not in (2, 61)" results in the following error (and similarly for detector: 61 if you don't skip that one):

  raise FileNotFoundError( FileNotFoundError: Not enough datasets (0) found for non-optional connection isr.flat (cpFlat) with minimum=1 for quantum data ID {instrument: 'DECam', detector: 2, exposure: 410915, ...}. 

Investigate this error, and either add the missing data to allow these detectors to process fully, or remove the files that are causing the gen3 pipeline to try to run ISR on these detectors so that the default ap_verify command can succeed without the additional query.

#### Activity

Hide
Meredith Rawls added a comment -

While I agree the behavior is not ideal, I'm not surprised by it.

DECam detectors 2 and 61 are at least sometimes bad. Depending on epoch, calibs for these detectors may or may not exist, and science images may or may not exist. This specific situation appears to be a mismatch. When processing DECam data, I either always expect these detectors to fail in some stage of processing, or I exclude them with a data query like in your example.

Show
Meredith Rawls added a comment - While I agree the behavior is not ideal, I'm not surprised by it. DECam detectors 2 and 61 are at least sometimes bad. Depending on epoch, calibs for these detectors may or may not exist, and science images may or may not exist. This specific situation appears to be a mismatch. When processing DECam data, I either always expect these detectors to fail in some stage of processing, or I exclude them with a data query like in your example.
Hide
Krzysztof Findeisen added a comment -

FWIW, the reason I asked for this issue to be investigated further is because butler query-datasets insists that raws and calibs exist for both detectors 2 and 61, and it looks like there's a good-sized selection of calibs to choose from. So something seems fishy to me.

In any case, the CI datasets do not have detectors 2 and 61 and do not try to run them, so we should be able to likewise remove these detectors from the full dataset if there's no other way to fix things.

Show
Krzysztof Findeisen added a comment - FWIW, the reason I asked for this issue to be investigated further is because butler query-datasets insists that raws and calibs exist for both detectors 2 and 61, and it looks like there's a good-sized selection of calibs to choose from. So something seems fishy to me. In any case, the CI datasets do not have detectors 2 and 61 and do not try to run them, so we should be able to likewise remove these detectors from the full dataset if there's no other way to fix things.

#### People

Assignee:
Unassigned
Reporter:
John Parejko
Watchers:
Eric Bellm, Ian Sullivan, John Parejko, Krzysztof Findeisen, Meredith Rawls