Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-29059

Investigate MJD-OBS complaints from LATISS translator

    XMLWordPrintable

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: obs_lsst
    • Labels:
      None
    • Story Points:
      1
    • Team:
      Architecture
    • Urgent?:
      No

      Description

      Merlin Fisher-Levine reports that metadata translation for modern LATISS data is issuing a lot of warnings about MJD-OBS not being found.

      Work out why this is happening and fix the problem.

        Attachments

          Activity

          Hide
          tjenness Tim Jenness added a comment -

          For example AT_20210217_000399

          Show
          tjenness Tim Jenness added a comment - For example AT_20210217_000399
          Hide
          tjenness Tim Jenness added a comment -

          By looking at the code everything points to this being triggered by the datetime_begin translation method. That assumes that the MJD-OBS header is present and will fail if it is missing. This failure causes cascading failures in header fixup and property calculations because we have many translation rules that critically depend on knowing the date of observation so as to understand what the situation is.

          I'm not sure I understand how MJD-OBS can be missing from a raw file but I'd have to look at all the data and see whether we do have some files with it missing. If they have a DATE-OBS then the code could be changed to look for both DATE-OBS and MJD-OBS and that might help.

          Show
          tjenness Tim Jenness added a comment - By looking at the code everything points to this being triggered by the datetime_begin translation method. That assumes that the MJD-OBS header is present and will fail if it is missing. This failure causes cascading failures in header fixup and property calculations because we have many translation rules that critically depend on knowing the date of observation so as to understand what the situation is. I'm not sure I understand how MJD-OBS can be missing from a raw file but I'd have to look at all the data and see whether we do have some files with it missing. If they have a DATE-OBS then the code could be changed to look for both DATE-OBS and MJD-OBS and that might help.
          Hide
          tjenness Tim Jenness added a comment -

          I've put a possible fix onto the ticket branch. It now explicitly checks for MJD-OBS and then falls back to DATE-OBS if MJD-OBS is missing.

          Show
          tjenness Tim Jenness added a comment - I've put a possible fix onto the ticket branch. It now explicitly checks for MJD-OBS and then falls back to DATE-OBS if MJD-OBS is missing.
          Hide
          mfisherlevine Merlin Fisher-Levine added a comment -

          Sorry, this was filed in a hurry, plus I hadn't had a chance to dig much. The problem isn't so much the warning itself (as you say, we really need that written), but the fact that it's spurious.

          Having just had a little poke, I've found the following:

          butler.get('raw'.... seems to not have that warning, but doing just about any other data product (bias, quickLookExp, etc) all give that warning, but when I look in their headers (fitsheader <filename> -k MJD_OBS -e 0) it seems to be there just fine, so there's something about the way that the raws are being read that works, but doesn't work for just about all other data types. I suspect this is something to do with the dreaded "composite" stuff, and may be Gen2 specific? Really not sure about that last bit though...

          Show
          mfisherlevine Merlin Fisher-Levine added a comment - Sorry, this was filed in a hurry, plus I hadn't had a chance to dig much. The problem isn't so much the warning itself (as you say, we really need that written), but the fact that it's spurious. Having just had a little poke, I've found the following: butler.get('raw'.... seems to not have that warning, but doing just about any other data product (bias, quickLookExp, etc) all give that warning, but when I look in their headers (fitsheader <filename> -k MJD_OBS -e 0) it seems to be there just fine, so there's something about the way that the raws are being read that works, but doesn't work for just about all other data types. I suspect this is something to do with the dreaded "composite" stuff, and may be Gen2 specific? Really not sure about that last bit though...
          Hide
          mfisherlevine Merlin Fisher-Levine added a comment -

          MWE:

          import lsst.daf.persistence as dafPersist
          butler = dafPersist.Butler('/project/shared/auxTel/rerun/quickLook')
          dataId = {'dayObs': '2021-03-11', 'seqNum': 300}
          butler.get('quickLookExp', **dataId)

          will give the error, and changing the quickLookExp to raw will make it go away. Checking that file, MJD-OBS is in the primary header.

          Show
          mfisherlevine Merlin Fisher-Levine added a comment - MWE: import lsst.daf.persistence as dafPersist butler = dafPersist.Butler('/project/shared/auxTel/rerun/quickLook') dataId = {'dayObs': '2021-03-11', 'seqNum': 300} butler.get('quickLookExp', **dataId) will give the error, and changing the quickLookExp to raw will make it go away. Checking that file, MJD-OBS is in the primary header.
          Hide
          tjenness Tim Jenness added a comment -

          Okay. I wasn't expecting header fix up to run on non-raws so it's possible that's some issue with us adding the fix_header call into a generic Exposure reader in gen2 butler.

          Hopefully this will be fixed automatically by DM-29187 (assuming we propagate headers properly from raws to calexp).

          Show
          tjenness Tim Jenness added a comment - Okay. I wasn't expecting header fix up to run on non-raws so it's possible that's some issue with us adding the fix_header call into a generic Exposure reader in gen2 butler. Hopefully this will be fixed automatically by DM-29187 (assuming we propagate headers properly from raws to calexp).
          Hide
          tjenness Tim Jenness added a comment -

          I'm going to mark this as done because I think it's an outcome of double fixing and that is no longer allowed (and the fixing only happens for raws now in gen3).

          Show
          tjenness Tim Jenness added a comment - I'm going to mark this as done because I think it's an outcome of double fixing and that is no longer allowed (and the fixing only happens for raws now in gen3).

            People

            Assignee:
            tjenness Tim Jenness
            Reporter:
            tjenness Tim Jenness
            Watchers:
            Merlin Fisher-Levine, Tim Jenness
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved:

                Jenkins Builds

                No builds found.