Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-27339

BOT has unrecognised physical_filters

    XMLWordPrintable

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: obs_lsst
    • Labels:
      None
    • Team:
      Data Release Production
    • Urgent?:
      No

      Description

      The infusion of BOT data at NCSA is failing due to physical_filters UNKNOWN~ND_OD0.3
      and UNKNOWN~ND_OD1.0 for data taken during c. 1 week.

      In each case Tony Johnson thinks that the FILTER2 key (the ND) is incorrect, so please map these files to UNKNOWN.

        Attachments

          Issue Links

            Activity

            Hide
            rhl Robert Lupton added a comment -

            The file /lsstdata/offline/teststand/BOT/storage/20200818/MC_C_20200818_000062/MC_C_20200818_000062_R11_S11.fits

            currently generates an error

            ingest WARN: Exposure LSSTCam:MC_C_20200818_000062 could not be registered: (sqlite3.IntegrityError) FOREIGN KEY constraint failed
            [SQL: INSERT INTO exposure (instrument, id, physical_filter, name, exposure_time, dark_time, observation_type, observation_reason, group_name, group_id, target_name, science_program, tracking_ra, tracking_dec, sky_angle, zenith_angle, timespan_begin, timespan_end) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?) ON CONFLICT DO NOTHING]
            [parameters: ('LSSTCam', 3020081800062, 'UNKNOWN~ND_OD0.3', 'MC_C_20200818_000062', 0.0, 0.0, 'bias', 'bias', '3020081800062', 3020081800062, 'UNKNOWN', '12475', None, None, None, None, 1597797403178997338, 1597797403178997338)]
            3178997338)]
            (Background on this error at: http://sqlalche.me/e/13/gkpj)

            Show
            rhl Robert Lupton added a comment - The file /lsstdata/offline/teststand/BOT/storage/20200818/MC_C_20200818_000062/MC_C_20200818_000062_R11_S11.fits currently generates an error ingest WARN: Exposure LSSTCam:MC_C_20200818_000062 could not be registered: (sqlite3.IntegrityError) FOREIGN KEY constraint failed [SQL: INSERT INTO exposure (instrument, id, physical_filter, name, exposure_time, dark_time, observation_type, observation_reason, group_name, group_id, target_name, science_program, tracking_ra, tracking_dec, sky_angle, zenith_angle, timespan_begin, timespan_end) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?) ON CONFLICT DO NOTHING] [parameters: ('LSSTCam', 3020081800062, 'UNKNOWN~ND_OD0.3', 'MC_C_20200818_000062', 0.0, 0.0, 'bias', 'bias', '3020081800062', 3020081800062, 'UNKNOWN', '12475', None, None, None, None, 1597797403178997338, 1597797403178997338)] 3178997338)] (Background on this error at: http://sqlalche.me/e/13/gkpj )
            Hide
            rhl Robert Lupton added a comment -

            Just a few lines.  Verified with 

            $ butler create myrepo-DM-27339
            $ butler register-instrument myrepo-DM-27339 lsst.obs.lsst.LsstCam
            $ butler ingest-raws myrepo-DM-27339 /lsstdata/offline/teststand/BOT/storage/20200818/MC_C_20200818_000062/MC_C_20200818_000062_R11_S11.fits
            lsst.obs.lsst.translators.lsst WARN: MC_C_20200818_000062: Unable to determine dark time. Setting from exposure time.
            ingest INFO: Successfully extracted metadata from 1 file with 0 failures
            ingest INFO: Exposure LSSTCam:MC_C_20200818_000062 ingested successfully
            ingest INFO: Successfully processed data from 1 exposure with 0 failures from exposure registration and 0 failures from file ingest.
            ingest INFO: Ingested 1 distinct Butler dataset 

            Show
            rhl Robert Lupton added a comment - Just a few lines.  Verified with  $ butler create myrepo-DM- 27339 $ butler register-instrument myrepo-DM- 27339 lsst.obs.lsst.LsstCam $ butler ingest-raws myrepo-DM- 27339 /lsstdata/offline/teststand/BOT/storage/ 20200818 /MC_C_20200818_000062/MC_C_20200818_000062_R11_S11.fits lsst.obs.lsst.translators.lsst WARN: MC_C_20200818_000062: Unable to determine dark time. Setting from exposure time. ingest INFO: Successfully extracted metadata from 1 file with 0 failures ingest INFO: Exposure LSSTCam:MC_C_20200818_000062 ingested successfully ingest INFO: Successfully processed data from 1 exposure with 0 failures from exposure registration and 0 failures from file ingest. ingest INFO: Ingested 1 distinct Butler dataset
            Hide
            tjenness Tim Jenness added a comment - - edited

            Another approach would be to do the fix in the ComCam fix_header method and blanking the FILTER2 keyword if FILTER is blank. That would be a legitimate fix since we know that FILTER2 is meant to be blank in this situation. In fact I see that in this case there is no FILTER header at all so something like:

            if "FILTER" not in header:
                header["FILTER"] = None
                header["FILTER2"] = None
                modified = True
                log.debug(...)
            

            would work.

            Show
            tjenness Tim Jenness added a comment - - edited Another approach would be to do the fix in the ComCam fix_header method and blanking the FILTER2 keyword if FILTER is blank. That would be a legitimate fix since we know that FILTER2 is meant to be blank in this situation. In fact I see that in this case there is no FILTER header at all so something like: if "FILTER" not in header: header["FILTER"] = None header["FILTER2"] = None modified = True log.debug(...) would work.
            Hide
            rhl Robert Lupton added a comment -

            I think I prefer my fix, as Tony did write the FILTER2 keyword, even though he now regrets it.

            Show
            rhl Robert Lupton added a comment - I think I prefer my fix, as Tony did write the FILTER2 keyword, even though he now regrets it.
            Hide
            tjenness Tim Jenness added a comment -

            The ticket states that the header is known to be wrong. The purpose of the "fix_header" method is to correct known header problems before the metadata translator sees them. This has two advantages:

            • People can now retrieve the header and get what was meant to be written
            • The metadata translator has to do more work (in this case two equality checks for every single header rather than one in check for every header.

            Since we aren't going to take a lot of ComCam data it doesn't really matter, but for LATISS and LSSTCam it will matter that we can fix headers in a targeted way rather than having the translator continually wonder about 2020 data headers when it's translating survey headers in 2030.

            Show
            tjenness Tim Jenness added a comment - The ticket states that the header is known to be wrong. The purpose of the "fix_header" method is to correct known header problems before the metadata translator sees them. This has two advantages: People can now retrieve the header and get what was meant to be written The metadata translator has to do more work (in this case two equality checks for every single header rather than one in check for every header. Since we aren't going to take a lot of ComCam data it doesn't really matter, but for LATISS and LSSTCam it will matter that we can fix headers in a targeted way rather than having the translator continually wonder about 2020 data headers when it's translating survey headers in 2030.
            Hide
            rhl Robert Lupton added a comment -

            OK, I'll defer to Tim's request. Stay tuned for another review request

            Show
            rhl Robert Lupton added a comment - OK, I'll defer to Tim's request. Stay tuned for another review request
            Hide
            rhl Robert Lupton added a comment -
            Show
            rhl Robert Lupton added a comment - Take 2. https://github.com/lsst/obs_lsst/pull/266
            Hide
            mfisherlevine Merlin Fisher-Levine added a comment -

            Comment on the log message, otherwise all looks good.

            Show
            mfisherlevine Merlin Fisher-Levine added a comment - Comment on the log message, otherwise all looks good.

              People

              Assignee:
              rhl Robert Lupton
              Reporter:
              rhl Robert Lupton
              Reviewers:
              Merlin Fisher-Levine
              Watchers:
              Merlin Fisher-Levine, Robert Lupton, Tim Jenness
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Jenkins Builds

                  No builds found.