Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-23629

Error on gen2 butler ingestion using NCSA Test stand data

    Details

    • Type: Story
    • Status: To Do
    • Resolution: Unresolved
    • Fix Version/s: None
    • Component/s: pipe_tasks
    • Labels:
      None
    • Templates:
    • Team:
      Architecture
    • Urgent?:
      No

      Description

      I found a bug in the gen2 butler ingestion of data from the NCSA test stand. This has not occurred at CP.

      The files in question are available in lsst-dev:/scratch/srp/gen2bug/

      To recreate the error:

      1. mkdir /tmp/repo
      2. echo lsst.obs.lsst.latiss.LatissMapper >/tmp/repo/_mapper
      3. scp lsst-dev.ncsa.illinois.edu:/shared/srp/gen2_bug/*fits /tmp
      4. setup the DM software stack to use the gen2 butler
      5. run the following:

      $ cat foo.py
      from lsst.pipe.tasks.ingest import IngestTask
       
      task = IngestTask.prepareTask("/tmp/repo")
      task.ingestFiles("/tmp/2020022500001-det000.fits")
      

      the output is the following:

      $ python foo.py
      /opt/lsst/software/stack/python/miniconda3-4.7.10/envs/lsst-scipipe-4d7b902/lib/python3.7/site-packages/astropy/config/configuration.py:532: ConfigurationMissingWarning: Configuration defaults will be used due to PermissionError:13 on None
        warn(ConfigurationMissingWarning(msg))
      root INFO: Loading config overrride file '/opt/lsst/software/stack/stack/miniconda3-4.7.10-4d7b902/Linux64/obs_lsst/19.0.0-22-g1f34043/config/ingest.py'
      root INFO: Loading config overrride file '/opt/lsst/software/stack/stack/miniconda3-4.7.10-4d7b902/Linux64/obs_lsst/19.0.0-22-g1f34043/config/latiss/ingest.py'
      LsstCamMapper WARN: Unable to find valid calib root directory
      CameraMapper INFO: Loading exposure registry from /tmp/repo/registry.sqlite3
      ingest.register INFO: Table "raw" exists.  Skipping creation
      lsst.obs.lsst.translators.latiss WARN: AT_O_20200225_000001: Unable to determine the filter
      lsst.obs.lsst.translators.latiss WARN: AT_O_20200225_000001: Dark time less than exposure time. Setting dark time to the exposure time.
      astro_metadata_translator.observationInfo WARN: Ignoring Error calculating property 'boresight_airmass' using translator <class 'lsst.obs.lsst.translators.latiss.LsstLatissTranslator'>: 'Unable to determine AltAz of science observation'
      astro_metadata_translator.observationInfo WARN: Ignoring Error calculating property 'boresight_rotation_angle' using translator <class 'lsst.obs.lsst.translators.latiss.LsstLatissTranslator'>: "Could not find ['ROTPA', 'ROTANGLE'] in header"
      astro_metadata_translator.observationInfo WARN: Ignoring Error calculating property 'object' using translator <class 'lsst.obs.lsst.translators.latiss.LsstLatissTranslator'>: "Could not find ['OBJECT'] in header"
      astro_metadata_translator.observationInfo WARN: Ignoring Error calculating property 'tracking_radec' using translator <class 'lsst.obs.lsst.translators.latiss.LsstLatissTranslator'>: 'Unable to determine tracking RA/Dec of science observation'
      astro_metadata_translator.observationInfo WARN: Ignoring Error calculating property 'altaz_begin' using translator <class 'lsst.obs.lsst.translators.latiss.LsstLatissTranslator'>: 'Unable to determine AltAz of science observation'
      ingest INFO: /tmp/2020022500001-det000.fits --<move>--> /tmp/repo/raw/2020-02-25/2020022500001-det000.fits
      ingest WARN: Failed to ingest file /tmp/2020022500001-det000.fits: Failed to register file /tmp/2020022500001-det000.fits
      Traceback (most recent call last):
        File "/opt/lsst/software/stack/stack/miniconda3-4.7.10-4d7b902/Linux64/pipe_tasks/19.0.0-23-g5d8da22d+1/python/lsst/pipe/tasks/ingest.py", line 585, in runFile
          self.register.addRow(registry, info, dryrun=args.dryrun, create=args.create)
        File "/opt/lsst/software/stack/stack/miniconda3-4.7.10-4d7b902/Linux64/pipe_tasks/19.0.0-23-g5d8da22d+1/python/lsst/pipe/tasks/ingest.py", line 348, in addRow
          values = [self.typemap[tt](info[col]) for col, tt in self.config.columns.items()]
        File "/opt/lsst/software/stack/stack/miniconda3-4.7.10-4d7b902/Linux64/pipe_tasks/19.0.0-23-g5d8da22d+1/python/lsst/pipe/tasks/ingest.py", line 348, in <listcomp>
          values = [self.typemap[tt](info[col]) for col, tt in self.config.columns.items()]
      KeyError: 'object'
       
      The above exception was the direct cause of the following exception:
       
      Traceback (most recent call last):
        File "/opt/lsst/software/stack/stack/miniconda3-4.7.10-4d7b902/Linux64/pipe_tasks/19.0.0-23-g5d8da22d+1/python/lsst/pipe/tasks/ingest.py", line 599, in run
          self.runFile(infile, registry, args, pos)
        File "/opt/lsst/software/stack/stack/miniconda3-4.7.10-4d7b902/Linux64/pipe_tasks/19.0.0-23-g5d8da22d+1/python/lsst/pipe/tasks/ingest.py", line 587, in runFile
          raise IngestError(f"Failed to register file {infile}", infile, pos) from exc
      lsst.pipe.tasks.ingest.IngestError: Failed to register file /tmp/2020022500001-det000.fits
       
      The above exception was the direct cause of the following exception:
       
      Traceback (most recent call last):
        File "foo.py", line 5, in <module>
          task.ingestFiles("/tmp/2020022500001-det000.fits")
        File "/opt/lsst/software/stack/stack/miniconda3-4.7.10-4d7b902/Linux64/pipe_tasks/19.0.0-23-g5d8da22d+1/python/lsst/pipe/tasks/ingest.py", line 621, in ingestFiles
          self.run(self._args)
        File "/opt/lsst/software/stack/stack/miniconda3-4.7.10-4d7b902/Linux64/pipe_tasks/19.0.0-23-g5d8da22d+1/python/lsst/pipe/tasks/ingest.py", line 603, in run
          raise IngestError(f"Failed to ingest file {infile}", infile, pos) from exc
      lsst.pipe.tasks.ingest.IngestError: Failed to ingest file /tmp/2020022500001-det000.fits
      

      Beyond the ingestion failure, there's another issue where the file is moved to the new location in /tmp/repo/raw even though the file is bad.
      I ran into this issue with the OODS, because I expected that once I was informed the ingestion failed, I would be able to move the file from it's original
      location into a new directory, "badfiles", so they could be looked at later. I expected that since the butler rejected it, it would have been left in
      place so the OODS could deal with it as it saw fit.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                spietrowicz Steve Pietrowicz
                Watchers:
                Felipe Menanteau, Kian-Tat Lim, Michelle Gower, Steve Pietrowicz, Tim Jenness
              • Votes:
                0 Vote for this issue
                Watchers:
                5 Start watching this issue

                Dates

                • Created:
                  Updated:

                  Summary Panel