Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-18748

Missing ExpTime in auxTel teststand images cause ingestImages.py to abort

    Details

    • Type: Bug
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: obs_lsst
    • Labels:
      None

      Description

      3 auxtel teststand images from 03/29/2019 failed to ingest with the following output:

      $ ingestImages.py /project/production/tmpdataloc/auxTel/gen2repo --calib /project/production/tmpdataloc/auxTel/gen2repo/CALIB --ignore-ingested --doraise /project/production/tmpdataloc/auxTel/storage/2019-03-29/AT_O_20190329_000022-ats-wfs_ccd.fits
      root INFO: Loading config overrride file '/software/lsstsw/stack_20190330/stack/miniconda3-4.5.12-1172c30/Linux64/obs_lsst/17.0.1-8-g5a63230+6/config/ingest.py'
      root INFO: Loading config overrride file '/software/lsstsw/stack_20190330/stack/miniconda3-4.5.12-1172c30/Linux64/obs_lsst/17.0.1-8-g5a63230+6/config/auxTel/ingest.py'
      CameraMapper INFO: Loading exposure registry from /project/production/tmpdataloc/auxTel/gen2repo/registry.sqlite3
      astro_metadata_translator.observationInfo WARN: Ignoring Error calculating property 'datetime_end' using translator <class 'lsst.obs.lsst.translators.auxTel.LsstAuxTelTranslator'>: "Could not find ['EXPTIME'] in header"
      astro_metadata_translator.observationInfo WARN: Ignoring Error calculating property 'exposure_time' using translator <class 'lsst.obs.lsst.translators.auxTel.LsstAuxTelTranslator'>: "Could not find ['EXPTIME'] in header"
      astro_metadata_translator.observationInfo WARN: Ignoring Error calculating property 'dark_time' using translator <class 'lsst.obs.lsst.translators.auxTel.LsstAuxTelTranslator'>: "Could not find ['DARKTIME', 'EXPTIME'] in header"
      astro_metadata_translator.observationInfo WARN: Ignoring Error calculating property 'boresight_airmass' using translator <class 'lsst.obs.lsst.translators.auxTel.LsstAuxTelTranslator'>: "Could not find ['AMSTART'] in header"
      astro_metadata_translator.observationInfo WARN: Ignoring Error calculating property 'boresight_rotation_angle' using translator <class 'lsst.obs.lsst.translators.auxTel.LsstAuxTelTranslator'>: "Could not find ['ROTANGLE'] in header"
      astro_metadata_translator.observationInfo WARN: Ignoring Error calculating property 'object' using translator <class 'lsst.obs.lsst.translators.auxTel.LsstAuxTelTranslator'>: "Could not find ['OBJECT'] in header"
      astro_metadata_translator.observationInfo WARN: Ignoring Error calculating property 'observation_type' using translator <class 'lsst.obs.lsst.translators.auxTel.LsstAuxTelTranslator'>: "Could not find ['EXPTIME'] in header"
      ingest.parse WARN: translate_expTime failed to translate expTime: 'NoneType' object has no attribute 'value'
      ingest.parse WARN: translate_imageType failed to translate imageType: 'NoneType' object has no attribute 'upper'
      ingest INFO: /project/production/tmpdataloc/auxTel/storage/2019-03-29/AT_O_20190329_000022-ats-wfs_ccd.fits --<link>--> /project/production/tmpdataloc/auxTel/gen2repo/raw/2019-03-29/2019032900022-det000.fits
      Traceback (most recent call last):
        File "/software/lsstsw/stack_20190330/stack/miniconda3-4.5.12-1172c30/Linux64/pipe_tasks/17.0.1-5-gf0ac6446+8/bin/ingestImages.py", line 3, in <module>
          IngestTask.parseAndRun()
        File "/software/lsstsw/stack_20190330/stack/miniconda3-4.5.12-1172c30/Linux64/pipe_tasks/17.0.1-5-gf0ac6446+8/python/lsst/pipe/tasks/ingest.py", line 416, in parseAndRun
          task.run(args)
        File "/software/lsstsw/stack_20190330/stack/miniconda3-4.5.12-1172c30/Linux64/pipe_tasks/17.0.1-5-gf0ac6446+8/python/lsst/pipe/tasks/ingest.py", line 553, in run
          self.register.addRow(registry, info, dryrun=args.dryrun, create=args.create)
        File "/software/lsstsw/stack_20190330/stack/miniconda3-4.5.12-1172c30/Linux64/pipe_tasks/17.0.1-5-gf0ac6446+8/python/lsst/pipe/tasks/ingest.py", line 357, in addRow
          values = [self.typemap[tt](info[col]) for col, tt in self.config.columns.items()]
        File "/software/lsstsw/stack_20190330/stack/miniconda3-4.5.12-1172c30/Linux64/pipe_tasks/17.0.1-5-gf0ac6446+8/python/lsst/pipe/tasks/ingest.py", line 357, in <listcomp>
          values = [self.typemap[tt](info[col]) for col, tt in self.config.columns.items()]
      KeyError: 'expTime'
      

      translate_header.py -p lsst.obs.lsst.translators /project/production/tmpdataloc/auxTel/storage/2019-03-29/AT_O_20190329_000022-ats-wfs_ccd.fits
      Analyzing /project/production/tmpdataloc/auxTel/storage/2019-03-29/AT_O_20190329_000022-ats-wfs_ccd.fits...
      KeyError("Error calculating property 'datetime_end' using translator <class 'lsst.obs.lsst.translators.auxTel.LsstAuxTelTranslator'> and file /project/production/tmpdataloc/auxTel/storage/2019-03-29/AT_O_20190329_000022-ats-wfs_ccd.fits")
      Files with failed translations:
      	/project/production/tmpdataloc/auxTel/storage/2019-03-29/AT_O_20190329_000022-ats-wfs_ccd.fits
      

        Attachments

          Activity

          Hide
          mfisherlevine Merlin Fisher-Levine added a comment -

          I don't think the GROUPID as a dumping ground really matters right now, to be honest. If they dump the wrong stuff in there hopefully things will fall over, and until the missing keys are fixed I think most of this data is not really useful as-is, so I think that's fine. Let me take a quick glance at the PR again and confirm the merge...

          Show
          mfisherlevine Merlin Fisher-Levine added a comment - I don't think the GROUPID as a dumping ground really matters right now, to be honest. If they dump the wrong stuff in there hopefully things will fall over, and until the missing keys are fixed I think most of this data is not really useful as-is, so I think that's fine. Let me take a quick glance at the PR again and confirm the merge...
          Hide
          mfisherlevine Merlin Fisher-Levine added a comment -

          The astro_metadata_translator PR looks good to merge, all comments addressed.

          For obs_lsst: did you add comments about the order of things in test_keys? And did you add a warning when getting the darkTime from expTime?

          Show
          mfisherlevine Merlin Fisher-Levine added a comment - The astro_metadata_translator PR looks good to merge, all comments addressed. For obs_lsst : did you add comments about the order of things in test_keys? And did you add a warning when getting the darkTime from expTime ?
          Hide
          tjenness Tim Jenness added a comment -

          I did add the darktime warning. I didn't yet make the test keys even more explicit because I keep reading the big comment before the test_keys where I am very explicitly explaining the issue. But I'll try to think of a more explicit variable name.

          Show
          tjenness Tim Jenness added a comment - I did add the darktime warning. I didn't yet make the test keys even more explicit because I keep reading the big comment before the test_keys where I am very explicitly explaining the issue. But I'll try to think of a more explicit variable name.
          Hide
          mfisherlevine Merlin Fisher-Levine added a comment -

          Fair enough - merge when ready then

          Show
          mfisherlevine Merlin Fisher-Levine added a comment - Fair enough - merge when ready then
          Hide
          tjenness Tim Jenness added a comment -

          Thanks for the review. Merged.

          Show
          tjenness Tim Jenness added a comment - Thanks for the review. Merged.

            People

            • Assignee:
              tjenness Tim Jenness
              Reporter:
              mgower Michelle Gower
              Reviewers:
              Merlin Fisher-Levine
              Watchers:
              Felipe Menanteau, Merlin Fisher-Levine, Michelle Gower, Patrick Ingraham, Tim Jenness
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Summary Panel