# BOT has unrecognised physical_filters

XMLWordPrintable

#### Details

• Type: Story
• Status: Done
• Resolution: Done
• Fix Version/s: None
• Component/s:
• Labels:
None
• Team:
Data Release Production
• Urgent?:
No

#### Description

The infusion of BOT data at NCSA is failing due to physical_filters UNKNOWN~ND_OD0.3
and UNKNOWN~ND_OD1.0 for data taken during c. 1 week.

In each case Tony Johnson thinks that the FILTER2 key (the ND) is incorrect, so please map these files to UNKNOWN.

#### Activity

Hide
Robert Lupton added a comment -

The file /lsstdata/offline/teststand/BOT/storage/20200818/MC_C_20200818_000062/MC_C_20200818_000062_R11_S11.fits

currently generates an error

ingest WARN: Exposure LSSTCam:MC_C_20200818_000062 could not be registered: (sqlite3.IntegrityError) FOREIGN KEY constraint failed
[SQL: INSERT INTO exposure (instrument, id, physical_filter, name, exposure_time, dark_time, observation_type, observation_reason, group_name, group_id, target_name, science_program, tracking_ra, tracking_dec, sky_angle, zenith_angle, timespan_begin, timespan_end) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?) ON CONFLICT DO NOTHING]
[parameters: ('LSSTCam', 3020081800062, 'UNKNOWN~ND_OD0.3', 'MC_C_20200818_000062', 0.0, 0.0, 'bias', 'bias', '3020081800062', 3020081800062, 'UNKNOWN', '12475', None, None, None, None, 1597797403178997338, 1597797403178997338)]
3178997338)]
(Background on this error at: http://sqlalche.me/e/13/gkpj)

Show
Robert Lupton added a comment - The file /lsstdata/offline/teststand/BOT/storage/20200818/MC_C_20200818_000062/MC_C_20200818_000062_R11_S11.fits currently generates an error ingest WARN: Exposure LSSTCam:MC_C_20200818_000062 could not be registered: (sqlite3.IntegrityError) FOREIGN KEY constraint failed [SQL: INSERT INTO exposure (instrument, id, physical_filter, name, exposure_time, dark_time, observation_type, observation_reason, group_name, group_id, target_name, science_program, tracking_ra, tracking_dec, sky_angle, zenith_angle, timespan_begin, timespan_end) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?) ON CONFLICT DO NOTHING] [parameters: ('LSSTCam', 3020081800062, 'UNKNOWN~ND_OD0.3', 'MC_C_20200818_000062', 0.0, 0.0, 'bias', 'bias', '3020081800062', 3020081800062, 'UNKNOWN', '12475', None, None, None, None, 1597797403178997338, 1597797403178997338)] 3178997338)] (Background on this error at: http://sqlalche.me/e/13/gkpj )
Hide
Robert Lupton added a comment -

Just a few lines.  Verified with

 $butler create myrepo-DM-27339 $ butler register-instrument myrepo-DM-27339 lsst.obs.lsst.LsstCam $butler ingest-raws myrepo-DM-27339 /lsstdata/offline/teststand/BOT/storage/20200818/MC_C_20200818_000062/MC_C_20200818_000062_R11_S11.fits lsst.obs.lsst.translators.lsst WARN: MC_C_20200818_000062: Unable to determine dark time. Setting from exposure time. ingest INFO: Successfully extracted metadata from 1 file with 0 failures ingest INFO: Exposure LSSTCam:MC_C_20200818_000062 ingested successfully ingest INFO: Successfully processed data from 1 exposure with 0 failures from exposure registration and 0 failures from file ingest. ingest INFO: Ingested 1 distinct Butler dataset  Show Robert Lupton added a comment - Just a few lines. Verified with$ butler create myrepo-DM- 27339 $butler register-instrument myrepo-DM- 27339 lsst.obs.lsst.LsstCam$ butler ingest-raws myrepo-DM- 27339 /lsstdata/offline/teststand/BOT/storage/ 20200818 /MC_C_20200818_000062/MC_C_20200818_000062_R11_S11.fits lsst.obs.lsst.translators.lsst WARN: MC_C_20200818_000062: Unable to determine dark time. Setting from exposure time. ingest INFO: Successfully extracted metadata from 1 file with 0 failures ingest INFO: Exposure LSSTCam:MC_C_20200818_000062 ingested successfully ingest INFO: Successfully processed data from 1 exposure with 0 failures from exposure registration and 0 failures from file ingest. ingest INFO: Ingested 1 distinct Butler dataset
Hide
Tim Jenness added a comment - - edited

Another approach would be to do the fix in the ComCam fix_header method and blanking the FILTER2 keyword if FILTER is blank. That would be a legitimate fix since we know that FILTER2 is meant to be blank in this situation. In fact I see that in this case there is no FILTER header at all so something like:

 if "FILTER" not in header:  header["FILTER"] = None  header["FILTER2"] = None  modified = True  log.debug(...) 

would work.

Show
Tim Jenness added a comment - - edited Another approach would be to do the fix in the ComCam fix_header method and blanking the FILTER2 keyword if FILTER is blank. That would be a legitimate fix since we know that FILTER2 is meant to be blank in this situation. In fact I see that in this case there is no FILTER header at all so something like: if "FILTER" not in header: header["FILTER"] = None header["FILTER2"] = None modified = True log.debug(...) would work.
Hide
Robert Lupton added a comment -

I think I prefer my fix, as Tony did write the FILTER2 keyword, even though he now regrets it.

Show
Robert Lupton added a comment - I think I prefer my fix, as Tony did write the FILTER2 keyword, even though he now regrets it.
Hide
Tim Jenness added a comment -

The ticket states that the header is known to be wrong. The purpose of the "fix_header" method is to correct known header problems before the metadata translator sees them. This has two advantages:

• People can now retrieve the header and get what was meant to be written
• The metadata translator has to do more work (in this case two equality checks for every single header rather than one in check for every header.

Since we aren't going to take a lot of ComCam data it doesn't really matter, but for LATISS and LSSTCam it will matter that we can fix headers in a targeted way rather than having the translator continually wonder about 2020 data headers when it's translating survey headers in 2030.

Show
Hide
Robert Lupton added a comment -

OK, I'll defer to Tim's request. Stay tuned for another review request

Show
Robert Lupton added a comment - OK, I'll defer to Tim's request. Stay tuned for another review request
Hide
Robert Lupton added a comment -
Show
Robert Lupton added a comment - Take 2. https://github.com/lsst/obs_lsst/pull/266
Hide
Merlin Fisher-Levine added a comment -

Comment on the log message, otherwise all looks good.

Show
Merlin Fisher-Levine added a comment - Comment on the log message, otherwise all looks good.

#### People

Assignee:
Robert Lupton
Reporter:
Robert Lupton
Reviewers:
Merlin Fisher-Levine
Watchers:
Merlin Fisher-Levine, Robert Lupton, Tim Jenness