Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-19766

DECam instcals fail to process with invalid DateTime

    XMLWordPrintable

    Details

    • Story Points:
      2
    • Sprint:
      Arch 2019-05-20
    • Team:
      Architecture

      Description

      When a user ingests a DECam instcal and runs exposure.getVisitInfo().getDate(), it returns an empty DateTime. ProcessCcd naturally assumes a non-empty DateTime and fails with a "DateTime not valid" error.

      We think this problem originates in astro_metadata_translator, which likely was made to work with DECam raws and not DECam instcals. Instcals are apparently missing some of the expected header keywords.

      This ticket is to fix things so ingested instcals have proper DateTimes and to add a test (presumably in astro_metadata_translator) that checks an instcal header.

      See the full discussion on Slack here.

        Attachments

          Activity

          Hide
          tjenness Tim Jenness added a comment -

          I'll look at this when I'm back in Tucson next week. In the mean time can someone please tell me where to find a file so that I can look at the header.

          Show
          tjenness Tim Jenness added a comment - I'll look at this when I'm back in Tucson next week. In the mean time can someone please tell me where to find a file so that I can look at the header.
          Hide
          mrawls Meredith Rawls added a comment -

          Thanks Tim Jenness. I put an instcal on lsst-dev at /project/mrawls/c4d_190402_050618_ooi_VR_v1.fits.fz for you.

          Show
          mrawls Meredith Rawls added a comment - Thanks Tim Jenness . I put an instcal on lsst-dev at /project/mrawls/c4d_190402_050618_ooi_VR_v1.fits.fz for you.
          Hide
          tjenness Tim Jenness added a comment -

          If I run translate_header on that example file it's clear that the problem is that datetime_begin works but datetime_end is None and the code to calculate a VisitInfo is not expecting that to be the case. I need to add fallback code to derive datetime_end from begin and exposure_time.

          $ translate_header.py c4d_190402_050618_ooi_VR_v1.fits.fz 
          Analyzing c4d_190402_050618_ooi_VR_v1.fits.fz...
          instrument: DECam
          telescope: CTIO 4.0-m telescope
          datetime_begin: 2019-04-02T05:06:18.383
          altaz_begin: <AltAz Coordinate (obstime=2019-04-02T05:06:18.383, location=(1814299.43217096, -5214321.09237828, -3187415.56248863) m, pressure=0.0 hPa, temperature=0.0 deg_C, relative_humidity=0.0, obswl=1.0 micron): (az, alt) in deg
              (52.8675, 65.16)>
          boresight_airmass: 1.1
          boresight_rotation_angle: nan deg
          boresight_rotation_coord: unknown
          dark_time: 120.7646399 s
          datetime_end: None
          detector_exposure_id: 84529101
          detector_group: S
          detector_name: 29
          detector_num: 1
          detector_serial: S3-06_123195-15-3
          detector_unique_name: S29
          exposure_id: 845291
          exposure_time: 120.0 s
          location: (1814299.43217096, -5214321.09237828, -3187415.56248863) m
          object: 
          observation_id: ct4m20190402t050618
          observation_type: science
          physical_filter: VR DECam c0007 6300.0 2600.0
          pressure: 779.0 hPa
          relative_humidity: 38.0
          science_program: 2019A-0337
          temperature: 15.1 deg_C
          tracking_radec: <SkyCoord (FK5: equinox=J2000.000): (ra, dec) in deg
              (215.87037083, -13.60661111)>
          visit_id: 845291
          

          Show
          tjenness Tim Jenness added a comment - If I run translate_header on that example file it's clear that the problem is that datetime_begin works but datetime_end is None and the code to calculate a VisitInfo is not expecting that to be the case. I need to add fallback code to derive datetime_end from begin and exposure_time. $ translate_header.py c4d_190402_050618_ooi_VR_v1.fits.fz Analyzing c4d_190402_050618_ooi_VR_v1.fits.fz... instrument: DECam telescope: CTIO 4.0-m telescope datetime_begin: 2019-04-02T05:06:18.383 altaz_begin: <AltAz Coordinate (obstime=2019-04-02T05:06:18.383, location=(1814299.43217096, -5214321.09237828, -3187415.56248863) m, pressure=0.0 hPa, temperature=0.0 deg_C, relative_humidity=0.0, obswl=1.0 micron): (az, alt) in deg (52.8675, 65.16)> boresight_airmass: 1.1 boresight_rotation_angle: nan deg boresight_rotation_coord: unknown dark_time: 120.7646399 s datetime_end: None detector_exposure_id: 84529101 detector_group: S detector_name: 29 detector_num: 1 detector_serial: S3-06_123195-15-3 detector_unique_name: S29 exposure_id: 845291 exposure_time: 120.0 s location: (1814299.43217096, -5214321.09237828, -3187415.56248863) m object: observation_id: ct4m20190402t050618 observation_type: science physical_filter: VR DECam c0007 6300.0 2600.0 pressure: 779.0 hPa relative_humidity: 38.0 science_program: 2019A-0337 temperature: 15.1 deg_C tracking_radec: <SkyCoord (FK5: equinox=J2000.000): (ra, dec) in deg (215.87037083, -13.60661111)> visit_id: 845291
          Hide
          tjenness Tim Jenness added a comment -

          Simon Krughoff would you be able to do a short review? I fixed the date problem and then realized I had no tests that dates were being calculated so I added a couple of tests for that and had to fix SuprimeCam.

          Show
          tjenness Tim Jenness added a comment - Simon Krughoff would you be able to do a short review? I fixed the date problem and then realized I had no tests that dates were being calculated so I added a couple of tests for that and had to fix SuprimeCam.
          Hide
          krughoff Simon Krughoff added a comment -

          Done.  Looks good.

          Show
          krughoff Simon Krughoff added a comment - Done.  Looks good.

            People

            Assignee:
            tjenness Tim Jenness
            Reporter:
            mrawls Meredith Rawls
            Reviewers:
            Simon Krughoff
            Watchers:
            Hayden Smotherman, Kian-Tat Lim, Meredith Rawls, Simon Krughoff, Tim Jenness
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved: