Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-27281

comcam images from NTS will not ingest in gen3 butler

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: obs_lsst
    • Labels:
      None
    • Team:
      Data Release Production
    • Urgent?:
      No

      Description

      The images currently being generated on the NTS are failing to ingest into the gen3 butler (stack week 42).  In the discussion with Tim Jenness on slack, this is because the "r" (listed in the logs below) is not a valid physical filter.

       

      I don't know which other components this should be corrected in.

       

      Here are the logs from the OODS:

      INFO 2020-10-20 16:39:54.194Z ingest prep 402 : Successfully extracted metadata from 1 file with 0 failures
      WARNING 2020-10-20 16:39:54.200Z ingest warning 92 : Exposure LSSTComCam:CC_O_20201020_000015 could not be registered: (sqlite3.IntegrityError) FOREIGN KEY constraint failed
       [SQL: INSERT INTO exposure (instrument, id, physical_filter, name, exposure_time, dark_time, observation_type, observation_reason, group_name, group_id, target_name, science_program, tracking_ra, tracking_dec, sky_angle, zenith_angle, timespan_begin, timespan_end) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?) ON CONFLICT DO NOTHING]
       [parameters: ('LSSTComCam', 2020102000015, 'r', 'CC_O_20201020_000015', 2.0, 1.2261381149292, 'flat', 'flat', 'CALSET_20201020_1635', 4939948147756420020, 'UNKNOWN', 'unknown', None, None, None, None, 1603212049093504529, 1603212050159000000)]
       (Background on this error at: [http://sqlalche.me/e/13/gkpj])
       INFO 2020-10-20 16:39:54.200Z ingest run 554 : Successfully processed data from 0 exposures with 1 failure from exposure registration and 0 failures from file ingest.}}
       INFO 2020-10-20 16:39:54.200Z ingest run 558 : Ingested 0 distinct Butler datasets
       ERROR 2020-10-20 16:39:54.200Z lsst.ctrl.oods.fileIngester ingest_file 212 : gen3: /data/lsstdata/NTS/comcam/oods/gen3butler/staging/2020-10-20/CC_O_20201020_000015-R22S00.fits could not be ingested: None
      

        Attachments

          Issue Links

            Activity

            Hide
            tjenness Tim Jenness added a comment -

            "r" is a filter band and not a physical filter defined for ComCam. If this is a test stand without ComCam attached to it then it should be fixed to return either "empty" or a valid filter name.

            There metadata translation could be made to do fix these. Is there anything in the NTS files that tells the translator that it's a test stand? (I haven't looked). I'd also need to know what range of dates of NTS data the FILTER header is wrong for.

            Show
            tjenness Tim Jenness added a comment - "r" is a filter band and not a physical filter defined for ComCam. If this is a test stand without ComCam attached to it then it should be fixed to return either "empty" or a valid filter name. There metadata translation could be made to do fix these. Is there anything in the NTS files that tells the translator that it's a test stand? (I haven't looked). I'd also need to know what range of dates of NTS data the FILTER header is wrong for.
            Hide
            spietrowicz Steve Pietrowicz added a comment -

            I looked in the generated FITS files, and I didn't see anything that indicates the file was generated from the NTS.

             

            The files the OODS was attempting to ingest were from:

            2020-10-20T16:35:45.904 in CC_O_20201020_000001-R22S12.fits

            to 

            2020-10-20T18:57:46.612 in CC_O_20201020_000030-R22S22.fits

             

            Show
            spietrowicz Steve Pietrowicz added a comment - I looked in the generated FITS files, and I didn't see anything that indicates the file was generated from the NTS.   The files the OODS was attempting to ingest were from: 2020-10-20T16:35:45.904 in CC_O_20201020_000001-R22S12.fits to  2020-10-20T18:57:46.612 in CC_O_20201020_000030-R22S22.fits  
            Hide
            tjenness Tim Jenness added a comment -

            No hint of NTS in the files is a serious problem. I asked for this months ago because without it I can't tell whether this is a pretend ComCam image or a real ComCam image from the base and that makes it impossible for me to use "between these dates I need to fix the header like this". It also can lead to clashes in butler registries if you try to ingest real ComCam and NTS ComCam into the same repository.

            cc/ Felipe Menanteau

            Show
            tjenness Tim Jenness added a comment - No hint of NTS in the files is a serious problem. I asked for this months ago because without it I can't tell whether this is a pretend ComCam image or a real ComCam image from the base and that makes it impossible for me to use "between these dates I need to fix the header like this". It also can lead to clashes in butler registries if you try to ingest real ComCam and NTS ComCam into the same repository. cc/ Felipe Menanteau
            Hide
            tjenness Tim Jenness added a comment -

            Felipe Menanteau when did you hard code "r" into the header service? Was it hard-coded for NTS and real ComCam at the base? If so I might be able to convert "r" to "empty" for those date ranges.

            Show
            tjenness Tim Jenness added a comment - Felipe Menanteau when did you hard code "r" into the header service? Was it hard-coded for NTS and real ComCam at the base? If so I might be able to convert "r" to "empty" for those date ranges.
            Hide
            felipe Felipe Menanteau added a comment -

            Tim Jenness It was added at the end of June (June 23 commit), for the HeaderService regardless of location.
            https://github.com/lsst-dm/HeaderService/commit/c6ef1a9f395cd3cf193f01c649121a5c8a3cd064#diff-ef396b4236074c89b7b7c46397815aaa4c4313066a78241a79e3fcf89ad3bfa5

            We don't keep a record of when versions of HeaderService are deployed at the NTS/base, but FILTER=r is in HeaderService from version 2.3.1 onwards. You could use this for the logic to search for it.

            Show
            felipe Felipe Menanteau added a comment - Tim Jenness It was added at the end of June (June 23 commit), for the HeaderService regardless of location. https://github.com/lsst-dm/HeaderService/commit/c6ef1a9f395cd3cf193f01c649121a5c8a3cd064#diff-ef396b4236074c89b7b7c46397815aaa4c4313066a78241a79e3fcf89ad3bfa5 We don't keep a record of when versions of HeaderService are deployed at the NTS/base, but FILTER=r is in HeaderService from version 2.3.1 onwards. You could use this for the logic to search for it.
            Show
            rhl Robert Lupton added a comment - https://github.com/lsst/obs_lsst/pull/269
            Hide
            tjenness Tim Jenness added a comment -

            Some minor comments on the PR. To check that your filter list looks reasonable you can do:

            $ butler create mytest
            $ butler register-instrument mytest lsst.obs.lsst.LsstCam (or whatever)
            

            and then look at the mytest/gen3.sqlite3 file (easiest to do sqlite3 mytest/gen3.sqlite3 .dump | less and search for the filter strings.

            Show
            tjenness Tim Jenness added a comment - Some minor comments on the PR. To check that your filter list looks reasonable you can do: $ butler create mytest $ butler register-instrument mytest lsst.obs.lsst.LsstCam (or whatever) and then look at the mytest/gen3.sqlite3 file (easiest to do sqlite3 mytest/gen3.sqlite3 .dump | less and search for the filter strings.

              People

              Assignee:
              rhl Robert Lupton
              Reporter:
              spietrowicz Steve Pietrowicz
              Reviewers:
              Tim Jenness
              Watchers:
              Felipe Menanteau, Robert Lupton, Steve Pietrowicz, Tim Jenness
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Jenkins Builds

                  No builds found.