Details
-
Type:
Story
-
Status: Won't Fix
-
Resolution: Done
-
Fix Version/s: None
-
Component/s: ap_verify
-
Labels:
-
Epic Link:
-
Team:Alert Production
-
Urgent?:No
Description
Trying to process ap_verify_hits2015 with ap_verify using gen3 without specifying --data-query "detector not in (2, 61)" results in the following error (and similarly for detector: 61 if you don't skip that one):
raise FileNotFoundError(
|
FileNotFoundError: Not enough datasets (0) found for non-optional connection isr.flat (cpFlat) with minimum=1 for quantum data ID {instrument: 'DECam', detector: 2, exposure: 410915, ...}.
|
Investigate this error, and either add the missing data to allow these detectors to process fully, or remove the files that are causing the gen3 pipeline to try to run ISR on these detectors so that the default ap_verify command can succeed without the additional query.
Attachments
Issue Links
Activity
FWIW, the reason I asked for this issue to be investigated further is because butler query-datasets insists that raws and calibs exist for both detectors 2 and 61, and it looks like there's a good-sized selection of calibs to choose from. So something seems fishy to me.
In any case, the CI datasets do not have detectors 2 and 61 and do not try to run them, so we should be able to likewise remove these detectors from the full dataset if there's no other way to fix things.
Krzysztof Findeisen
added a comment - FWIW, the reason I asked for this issue to be investigated further is because butler query-datasets insists that raws and calibs exist for both detectors 2 and 61, and it looks like there's a good-sized selection of calibs to choose from. So something seems fishy to me.
In any case, the CI datasets do not have detectors 2 and 61 and do not try to run them, so we should be able to likewise remove these detectors from the full dataset if there's no other way to fix things. Meredith Rawls is this ticket still relevant?
It will become obsolete once DM-36477 is implemented.
Krzysztof Findeisen
added a comment - It will become obsolete once DM-36477 is implemented.
While I agree the behavior is not ideal, I'm not surprised by it.
DECam detectors 2 and 61 are at least sometimes bad. Depending on epoch, calibs for these detectors may or may not exist, and science images may or may not exist. This specific situation appears to be a mismatch. When processing DECam data, I either always expect these detectors to fail in some stage of processing, or I exclude them with a data query like in your example.