Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-17530

obs_lsst no longer supports TS3 test data

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: obs_lsst
    • Labels:
      None
    • Story Points:
      6
    • Sprint:
      Arch 2019-01-28, Arch 2019-02-04, Arch 2019-02-11
    • Team:
      Architecture

      Description

      Before the move to `astro_metadata_translator`, `obs_lsst` used to be able to usefully ingest the TS3 acceptance test data for the AuxTel chip (and others). I think it would be good to continue being able to ingest data in this format, as this is the canonical original state of the chips upon receipt by the project.

       

      Errors/warnings/failures thrown look like:

       

      Error calculating property 'visit_id' using translator <class 'lsst.obs.lsst.translators.auxTel.LsstAuxTelTranslator'>: 'DAYOBS'
      Unable to determine observation type. Guessing 'unknown'
      OrderedDict([('SIMPLE', True), ('BITPIX', 8), ('NAXIS', 0), ('EXTEND', True), ('ORIGIN', 'BNL'), ('DATE', '2016-07-22T03:33:55.918'), ('DATE-OBS', '2016-07-22T03:33:52.608'), ('MJD-OBS', 57591.14852555562), ('IMAGETAG', 'CCS'), ('TSTAND', 'BNL-TS3-2-Janeway'), ('INSTRUME', 'ArchonCCDController'), ('CONTROLL', 'ArchonCCDController'), ('CCD_MANU', 'ITL'), ('CCD_TYPE', 'STA3800C'), ('CCD_SERN', 'ITL-3800C-098'), ('LSST_NUM', 'ITL-3800C-098'), ('TESTTYPE', 'SFLAT_500'), ('IMGTYPE', 'FLAT'), ('SEQNUM', 0), ('TEMP_SET', -95.0), ('CCDTEMP', -94.78088), ('CCDBSS', -35.0), ('MONDIODE', -141.62008656027282), ('MONOWL', 500.015), ('PIXRATE', 549.4505494505494), ('FILTER', '275CutOn'), ('FILTPOS', 2), ('EXPTIME', 0.155), ('CTLRCFG', 'bnl_STA_20160718_B.acf'), ('FILENAME', 'ITL-3800C-098_sflat_500_flat_L025_20160721233355.fits'), ('DETSIZE', '[1:4072,1:4000]'), ('BINX', 1), ('BINY', 1), ('HEADVER', 1), ('CCDGAIN', 3.85), ('CCDNOISE', 1.0), ('CFGFILE', 'bnl_STA_20160718_B.acf'), ('CTLRID', '23723640293011484'), ('CTLRRV', 3), ('CTLRTYP', 1), ('CTLRVER', '1.0.762'), ('MONOCH-WAVELENG', 500.015), ('MONOCH-SLIT_A', 210.0), ('MONOCH-SLIT_B', 210.0), ('MONOCH-SLIT_C', 0.0), ('MONOCH-BANDPASS', 0.0), ('MONOCH-FILT_1', 2.0), ('MONOCH-MSTEPS', 1200.0), ('MONOCH-GRATING', 1.0), ('AMP0-IDN', 'KEITHLEY-INSTRUMENTS-INC.,MODEL-6487'), ('AMP0-AZERO', 'F'), ('AMP0-COUNT', 1246), ('AMP0-MEAS_NPLC', 0), ('AMP2-IDN', 'KEITHLEY-INSTRUMENTS-INC.,MODEL-6487'), ('AMP2-VOLTAGE', -35.0), ('AMP2-CURRENT', -70431.27), ('AMP2-ZERO_CHECK', 'off'), ('CHECKSUM', 'Z9Wbc9TaZ9Tab9Ta'), ('DATASUM', '0')])
      Error calculating property 'detector_exposure_id' using translator <class 'lsst.obs.lsst.translators.auxTel.LsstAuxTelTranslator'>: 'DAYOBS'
      ingest WARN: Failed to ingest file /datasets/auxTel/raw/BNL/prod/ITL-CCD/ITL-3800C-098/sflat_acq/v0/278/ITL-3800C-098_sflat_500_flat_L025_20160721233355.fits: 'visit'

       

      So it looks like there are several problems:

      1) Failure to calculate the `visit_id`

      2) Failure to calculate the `detector_exposure_id` (likely due to the above)

      3) Failure to correctly translate the observation type (we need both the testType and the imageType for this data - I realise this doesn't fit `astro_metadata_translator` model, but this is how the LSST acceptance testing works, and it should be supported somehow)

      4) Ingest currently just barfs a whole `OrderedDict` - I don't know if that's part of/due to one of the warnings/errors, or a normal ingest output. If the latter it should be removed, if the former then it should have some sort of introduction saying "XX failed to do YY with data OrderedDict" or something, because as it is, I don't know what that's apropos of, and so isn't useful.

      Output order seems to be

      • OrderedDict
      • Error calculating property 'exposure_id'
      • OrderedDict
      • Error calculating property 'visit_id'
      • Unable to determine observation type. Guessing 'bias'
      • OrderedDict
      • Error calculating property 'detector_exposure_id'
      • ingest WARN: Failed to ingest file

      The parser errors, because they don't use the LSST logger, aren't of the usual format either, so one can't tell at a glance which is an error, which is a warning etc.

        Attachments

          Activity

          Hide
          mfisherlevine Merlin Fisher-Levine added a comment -

          The ingest command can easily work if you just point it at that one file though, and the resulting parser should still be able to pick up the full path I think, so it might not be any work at all...

          Show
          mfisherlevine Merlin Fisher-Levine added a comment - The ingest command can easily work if you just point it at that one file though, and the resulting parser should still be able to pick up the full path I think, so it might not be any work at all...
          Hide
          krughoff Simon Krughoff added a comment -

          Agreed. I'm just saying the CI scripts I've been working on hard code the ingestion paths like this.

          Show
          krughoff Simon Krughoff added a comment - Agreed. I'm just saying the CI scripts I've been working on hard code the ingestion paths like this .
          Hide
          tjenness Tim Jenness added a comment -

          TS3 seems to be working now. I can ingest your example TS3 data at least.

          Show
          tjenness Tim Jenness added a comment - TS3 seems to be working now. I can ingest your example TS3 data at least.
          Hide
          mfisherlevine Merlin Fisher-Levine added a comment -

          Super minor comments, fine to merge whenever.

          Show
          mfisherlevine Merlin Fisher-Levine added a comment - Super minor comments, fine to merge whenever.
          Hide
          tjenness Tim Jenness added a comment -

          Merged. I tested on some of the TS3 data in /datasets/auxTel/BNL and it looks fine.

          Show
          tjenness Tim Jenness added a comment - Merged. I tested on some of the TS3 data in /datasets/auxTel/BNL and it looks fine.

            People

            • Assignee:
              tjenness Tim Jenness
              Reporter:
              mfisherlevine Merlin Fisher-Levine
              Reviewers:
              Merlin Fisher-Levine
              Watchers:
              Andrés Alejandro Plazas Malagón, John Swinbank, Merlin Fisher-Levine, Simon Krughoff, Tim Jenness
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Summary Panel