For example AT_20210217_000399

By looking at the code everything points to this being triggered by the datetime_begin translation method. That assumes that the MJD-OBS header is present and will fail if it is missing. This failure causes cascading failures in header fixup and property calculations because we have many translation rules that critically depend on knowing the date of observation so as to understand what the situation is.

I'm not sure I understand how MJD-OBS can be missing from a raw file but I'd have to look at all the data and see whether we do have some files with it missing. If they have a DATE-OBS then the code could be changed to look for both DATE-OBS and MJD-OBS and that might help.

I've put a possible fix onto the ticket branch. It now explicitly checks for MJD-OBS and then falls back to DATE-OBS if MJD-OBS is missing.

Sorry, this was filed in a hurry, plus I hadn't had a chance to dig much. The problem isn't so much the warning itself (as you say, we really need that written), but the fact that it's spurious.

Having just had a little poke, I've found the following:

butler.get('raw'.... seems to not have that warning, but doing just about any other data product (bias, quickLookExp, etc) all give that warning, but when I look in their headers (fitsheader <filename> -k MJD_OBS -e 0) it seems to be there just fine, so there's something about the way that the raws are being read that works, but doesn't work for just about all other data types. I suspect this is something to do with the dreaded "composite" stuff, and may be Gen2 specific? Really not sure about that last bit though...

MWE:

import lsst.daf.persistence as dafPersist

butler = dafPersist.Butler('/project/shared/auxTel/rerun/quickLook')

dataId = {'dayObs': '2021-03-11', 'seqNum': 300}

butler.get('quickLookExp', **dataId)

will give the error, and changing the quickLookExp to raw will make it go away. Checking that file, MJD-OBS is in the primary header.

Okay. I wasn't expecting header fix up to run on non-raws so it's possible that's some issue with us adding the fix_header call into a generic Exposure reader in gen2 butler.

Hopefully this will be fixed automatically by DM-29187 (assuming we propagate headers properly from raws to calexp).

I'm going to mark this as done because I think it's an outcome of double fixing and that is no longer allowed (and the fixing only happens for raws now in gen3).