# comcam images from NTS will not ingest in gen3 butler

XMLWordPrintable

#### Details

• Type: Bug
• Status: Done
• Resolution: Done
• Fix Version/s: None
• Component/s:
• Labels:
None
• Team:
Data Release Production
• Urgent?:
No

#### Description

The images currently being generated on the NTS are failing to ingest into the gen3 butler (stack week 42).  In the discussion with Tim Jenness on slack, this is because the "r" (listed in the logs below) is not a valid physical filter.

I don't know which other components this should be corrected in.

Here are the logs from the OODS:

 INFO 2020-10-20 16:39:54.194Z ingest prep 402 : Successfully extracted metadata from 1 file with 0 failures WARNING 2020-10-20 16:39:54.200Z ingest warning 92 : Exposure LSSTComCam:CC_O_20201020_000015 could not be registered: (sqlite3.IntegrityError) FOREIGN KEY constraint failed  [SQL: INSERT INTO exposure (instrument, id, physical_filter, name, exposure_time, dark_time, observation_type, observation_reason, group_name, group_id, target_name, science_program, tracking_ra, tracking_dec, sky_angle, zenith_angle, timespan_begin, timespan_end) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?) ON CONFLICT DO NOTHING]  [parameters: ('LSSTComCam', 2020102000015, 'r', 'CC_O_20201020_000015', 2.0, 1.2261381149292, 'flat', 'flat', 'CALSET_20201020_1635', 4939948147756420020, 'UNKNOWN', 'unknown', None, None, None, None, 1603212049093504529, 1603212050159000000)]  (Background on this error at: [http://sqlalche.me/e/13/gkpj])  INFO 2020-10-20 16:39:54.200Z ingest run 554 : Successfully processed data from 0 exposures with 1 failure from exposure registration and 0 failures from file ingest.}}  INFO 2020-10-20 16:39:54.200Z ingest run 558 : Ingested 0 distinct Butler datasets  ERROR 2020-10-20 16:39:54.200Z lsst.ctrl.oods.fileIngester ingest_file 212 : gen3: /data/lsstdata/NTS/comcam/oods/gen3butler/staging/2020-10-20/CC_O_20201020_000015-R22S00.fits could not be ingested: None 

#### Activity

Hide
Tim Jenness added a comment -

"r" is a filter band and not a physical filter defined for ComCam. If this is a test stand without ComCam attached to it then it should be fixed to return either "empty" or a valid filter name.

There metadata translation could be made to do fix these. Is there anything in the NTS files that tells the translator that it's a test stand? (I haven't looked). I'd also need to know what range of dates of NTS data the FILTER header is wrong for.

Show
Tim Jenness added a comment - "r" is a filter band and not a physical filter defined for ComCam. If this is a test stand without ComCam attached to it then it should be fixed to return either "empty" or a valid filter name. There metadata translation could be made to do fix these. Is there anything in the NTS files that tells the translator that it's a test stand? (I haven't looked). I'd also need to know what range of dates of NTS data the FILTER header is wrong for.
Hide
Steve Pietrowicz added a comment -

I looked in the generated FITS files, and I didn't see anything that indicates the file was generated from the NTS.

The files the OODS was attempting to ingest were from:

2020-10-20T16:35:45.904 in CC_O_20201020_000001-R22S12.fits

to

2020-10-20T18:57:46.612 in CC_O_20201020_000030-R22S22.fits

Show
Steve Pietrowicz added a comment - I looked in the generated FITS files, and I didn't see anything that indicates the file was generated from the NTS.   The files the OODS was attempting to ingest were from: 2020-10-20T16:35:45.904 in CC_O_20201020_000001-R22S12.fits to  2020-10-20T18:57:46.612 in CC_O_20201020_000030-R22S22.fits
Hide
Tim Jenness added a comment -

No hint of NTS in the files is a serious problem. I asked for this months ago because without it I can't tell whether this is a pretend ComCam image or a real ComCam image from the base and that makes it impossible for me to use "between these dates I need to fix the header like this". It also can lead to clashes in butler registries if you try to ingest real ComCam and NTS ComCam into the same repository.

Show
Tim Jenness added a comment - No hint of NTS in the files is a serious problem. I asked for this months ago because without it I can't tell whether this is a pretend ComCam image or a real ComCam image from the base and that makes it impossible for me to use "between these dates I need to fix the header like this". It also can lead to clashes in butler registries if you try to ingest real ComCam and NTS ComCam into the same repository. cc/ Felipe Menanteau
Hide
Tim Jenness added a comment -

Felipe Menanteau when did you hard code "r" into the header service? Was it hard-coded for NTS and real ComCam at the base? If so I might be able to convert "r" to "empty" for those date ranges.

Show
Tim Jenness added a comment - Felipe Menanteau when did you hard code "r" into the header service? Was it hard-coded for NTS and real ComCam at the base? If so I might be able to convert "r" to "empty" for those date ranges.
Hide
Felipe Menanteau added a comment -

Tim Jenness It was added at the end of June (June 23 commit), for the HeaderService regardless of location.

We don't keep a record of when versions of HeaderService are deployed at the NTS/base, but FILTER=r is in HeaderService from version 2.3.1 onwards. You could use this for the logic to search for it.

Show
Felipe Menanteau added a comment - Tim Jenness It was added at the end of June (June 23 commit), for the HeaderService regardless of location. https://github.com/lsst-dm/HeaderService/commit/c6ef1a9f395cd3cf193f01c649121a5c8a3cd064#diff-ef396b4236074c89b7b7c46397815aaa4c4313066a78241a79e3fcf89ad3bfa5 We don't keep a record of when versions of HeaderService are deployed at the NTS/base, but FILTER=r is in HeaderService from version 2.3.1 onwards. You could use this for the logic to search for it.
Hide
Robert Lupton added a comment -
Show
Robert Lupton added a comment - https://github.com/lsst/obs_lsst/pull/269
Hide
Tim Jenness added a comment -

Some minor comments on the PR. To check that your filter list looks reasonable you can do:

 $butler create mytest $ butler register-instrument mytest lsst.obs.lsst.LsstCam (or whatever) 

and then look at the mytest/gen3.sqlite3 file (easiest to do sqlite3 mytest/gen3.sqlite3 .dump | less and search for the filter strings.

Show
Tim Jenness added a comment - Some minor comments on the PR. To check that your filter list looks reasonable you can do: $butler create mytest$ butler register-instrument mytest lsst.obs.lsst.LsstCam (or whatever) and then look at the mytest/gen3.sqlite3 file (easiest to do sqlite3 mytest/gen3.sqlite3 .dump | less and search for the filter strings.

#### People

Assignee:
Robert Lupton
Reporter:
Steve Pietrowicz
Reviewers:
Tim Jenness
Watchers:
Felipe Menanteau, Robert Lupton, Steve Pietrowicz, Tim Jenness