Investigate MJD-OBS complaints from LATISS translator

XMLWordPrintable

Details

• Type: Story
• Status: Done
• Resolution: Done
• Fix Version/s: None
• Component/s:
• Labels:
None
• Story Points:
1
• Team:
Architecture
• Urgent?:
No

Description

Merlin Fisher-Levine reports that metadata translation for modern LATISS data is issuing a lot of warnings about MJD-OBS not being found.

Work out why this is happening and fix the problem.

Activity

Hide
Tim Jenness added a comment -

For example AT_20210217_000399

Show
Tim Jenness added a comment - For example AT_20210217_000399
Hide
Tim Jenness added a comment -

By looking at the code everything points to this being triggered by the datetime_begin translation method. That assumes that the MJD-OBS header is present and will fail if it is missing. This failure causes cascading failures in header fixup and property calculations because we have many translation rules that critically depend on knowing the date of observation so as to understand what the situation is.

I'm not sure I understand how MJD-OBS can be missing from a raw file but I'd have to look at all the data and see whether we do have some files with it missing. If they have a DATE-OBS then the code could be changed to look for both DATE-OBS and MJD-OBS and that might help.

Show
Tim Jenness added a comment - By looking at the code everything points to this being triggered by the datetime_begin translation method. That assumes that the MJD-OBS header is present and will fail if it is missing. This failure causes cascading failures in header fixup and property calculations because we have many translation rules that critically depend on knowing the date of observation so as to understand what the situation is. I'm not sure I understand how MJD-OBS can be missing from a raw file but I'd have to look at all the data and see whether we do have some files with it missing. If they have a DATE-OBS then the code could be changed to look for both DATE-OBS and MJD-OBS and that might help.
Hide
Tim Jenness added a comment -

I've put a possible fix onto the ticket branch. It now explicitly checks for MJD-OBS and then falls back to DATE-OBS if MJD-OBS is missing.

Show
Tim Jenness added a comment - I've put a possible fix onto the ticket branch. It now explicitly checks for MJD-OBS and then falls back to DATE-OBS if MJD-OBS is missing.
Hide
Merlin Fisher-Levine added a comment -

Sorry, this was filed in a hurry, plus I hadn't had a chance to dig much. The problem isn't so much the warning itself (as you say, we really need that written), but the fact that it's spurious.

Having just had a little poke, I've found the following:

butler.get('raw'.... seems to not have that warning, but doing just about any other data product (bias, quickLookExp, etc) all give that warning, but when I look in their headers (fitsheader <filename> -k MJD_OBS -e 0) it seems to be there just fine, so there's something about the way that the raws are being read that works, but doesn't work for just about all other data types. I suspect this is something to do with the dreaded "composite" stuff, and may be Gen2 specific? Really not sure about that last bit though...

Show
Hide
Merlin Fisher-Levine added a comment -

MWE:

 import lsst.daf.persistence as dafPersist butler = dafPersist.Butler('/project/shared/auxTel/rerun/quickLook') dataId = {'dayObs': '2021-03-11', 'seqNum': 300} butler.get('quickLookExp', **dataId)

will give the error, and changing the quickLookExp to raw will make it go away. Checking that file, MJD-OBS is in the primary header.

Show
Merlin Fisher-Levine added a comment - MWE: import lsst.daf.persistence as dafPersist butler = dafPersist.Butler('/project/shared/auxTel/rerun/quickLook') dataId = {'dayObs': '2021-03-11', 'seqNum': 300} butler.get('quickLookExp', **dataId) will give the error, and changing the quickLookExp to raw will make it go away. Checking that file, MJD-OBS is in the primary header.
Hide
Tim Jenness added a comment -

Okay. I wasn't expecting header fix up to run on non-raws so it's possible that's some issue with us adding the fix_header call into a generic Exposure reader in gen2 butler.

Hopefully this will be fixed automatically by DM-29187 (assuming we propagate headers properly from raws to calexp).

Show
Tim Jenness added a comment - Okay. I wasn't expecting header fix up to run on non-raws so it's possible that's some issue with us adding the fix_header call into a generic Exposure reader in gen2 butler. Hopefully this will be fixed automatically by DM-29187 (assuming we propagate headers properly from raws to calexp).
Hide
Tim Jenness added a comment -

I'm going to mark this as done because I think it's an outcome of double fixing and that is no longer allowed (and the fixing only happens for raws now in gen3).

Show
Tim Jenness added a comment - I'm going to mark this as done because I think it's an outcome of double fixing and that is no longer allowed (and the fixing only happens for raws now in gen3).

People

Assignee:
Tim Jenness
Reporter:
Tim Jenness
Watchers:
Merlin Fisher-Levine, Tim Jenness