Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-16554

Write metadata translators for obs_lsst cameras

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: obs_lsst
    • Labels:
      None
    • Story Points:
      10
    • Sprint:
      Arch 2018-11-12, Arch 2018-11-19, Arch 2018-11-26, Arch 2018-12-03, Arch 2018-12-10, Arch 2019-01-07, Arch 2019-01-14
    • Team:
      Architecture

      Description

      Add translators for cameras in obs_lsst. This is needed in order for LSST data files to be used with Gen 3 butler and also to allow a simple way for NCSA to extract metadata when ingesting test files into the data backbone.

        Attachments

          Issue Links

            Activity

            Hide
            tjenness Tim Jenness added a comment -

            Michelle Gower TS8 and PhoSim translations work now on the ticket branches. If you are writing Python to use the new translators you need to import lsst.obs.lsst.translators and that will register them with the translation system (with translate_header.py use the new --packages lsst.obs.lsst.translators option to make them available).

            I have not switched VisitInfo calculations to use these yet. I am currently trying to decide whether I override the ParseTask class from pipe_tasks to make it use the new translators rather than the old translators. If I do that then at least we won't be having to support the old and new translators and gen2 will be using the new ones as well.

            Show
            tjenness Tim Jenness added a comment - Michelle Gower TS8 and PhoSim translations work now on the ticket branches. If you are writing Python to use the new translators you need to import lsst.obs.lsst.translators and that will register them with the translation system (with translate_header.py use the new --packages lsst.obs.lsst.translators option to make them available). I have not switched VisitInfo calculations to use these yet. I am currently trying to decide whether I override the ParseTask class from pipe_tasks to make it use the new translators rather than the old translators. If I do that then at least we won't be having to support the old and new translators and gen2 will be using the new ones as well.
            Hide
            tjenness Tim Jenness added a comment -

            I am now at the point where I have translators for PhoSim, ImSim, AuxTel, and TS8 as they exist in Robert Lupton's test repository and the headers in obs_lsst.

            I have also cleaned up the WCS setting code so that it doesn't look for RATEL etc explicitly from the header again but gets the values from the VisitInfo that has already been calculated.

            Ideally I would replace the gen2 translator with the new translator and I can have a look at doing that whilst the review is ongoing. That would help remove the added overhead of supporting two translators.

            One remaining issue I have is that the IMSIM reduced calibration data files have no headers indicating what these data are (DM-16292): no instrument, telescope, or date, so it's difficult to translate the headers.

            This is quite a big ticket to review so can I suggest that Simon Krughoff handles astro_metadata_translator and obs_base and that the obs_lsst changes are shared by Merlin Fisher-Levine and James Chiang.

            It's possible that I've made mistakes in migrating logic from the gen2 butler translator.

            To test the translations from the command line you can use the translatehdr.py command:

            %  translate_header.py  -p lsst.obs.lsst.translators data/input/phosim
            Analyzing data/input/phosim/raw/204595/R11/00204595-R11-S20-det042-000.fits...
            instrument: PhoSim
            telescope: LSST
            datetime_begin: 2022-10-05T06:53:11.357
            altaz_begin: <AltAz Coordinate (obstime=2022-10-05T06:53:11.357, location=(1822838.3011943279, -1062819.3254546772, -6001635.768663893) m, pressure=0.0 hPa, temperature=0.0 deg_C, relative_humidity=0.0, obswl=1.0 micron): (az, alt) in deg
                (93.26675455, 81.72731083)>
            boresight_airmass: 1.01051444909673
            boresight_rotation_angle: 195.624333002684 deg
            boresight_rotation_coord: sky
            dark_time: 30.0 s
            datetime_end: 2022-10-05T06:53:41.357
            detector_exposure_id: 40919101
            detector_group: R11
            detector_name: S20
            detector_num: 101
            detector_serial: R11_S20
            exposure_id: 204595
            exposure_time: 30.0 s
            location: (1822838.3011943279, -1062819.3254546772, -6001635.768663893) m
            object: unknown
            observation_id: 204595
            observation_type: science
            physical_filter: i
            pressure: 520.0 mmHg
            relative_humidity: 40.0
            science_program: 204595
            temperature: 20.0 deg_C
            tracking_radec: <SkyCoord (ICRS): (ra, dec) in deg
                (55.67759886, -30.44239357)>
            visit_id: 204595
            

            (this requires both the astro_metadata_translator and obs_lsst branches to be set up).

            Thanks.

            Show
            tjenness Tim Jenness added a comment - I am now at the point where I have translators for PhoSim, ImSim, AuxTel, and TS8 as they exist in Robert Lupton 's test repository and the headers in obs_lsst. I have also cleaned up the WCS setting code so that it doesn't look for RATEL etc explicitly from the header again but gets the values from the VisitInfo that has already been calculated. Ideally I would replace the gen2 translator with the new translator and I can have a look at doing that whilst the review is ongoing. That would help remove the added overhead of supporting two translators. One remaining issue I have is that the IMSIM reduced calibration data files have no headers indicating what these data are ( DM-16292 ): no instrument, telescope, or date, so it's difficult to translate the headers. This is quite a big ticket to review so can I suggest that Simon Krughoff handles astro_metadata_translator and obs_base and that the obs_lsst changes are shared by Merlin Fisher-Levine and James Chiang . It's possible that I've made mistakes in migrating logic from the gen2 butler translator. To test the translations from the command line you can use the translatehdr.py command: % translate_header.py -p lsst.obs.lsst.translators data/input/phosim Analyzing data/input/phosim/raw/204595/R11/00204595-R11-S20-det042-000.fits... instrument: PhoSim telescope: LSST datetime_begin: 2022-10-05T06:53:11.357 altaz_begin: <AltAz Coordinate (obstime=2022-10-05T06:53:11.357, location=(1822838.3011943279, -1062819.3254546772, -6001635.768663893) m, pressure=0.0 hPa, temperature=0.0 deg_C, relative_humidity=0.0, obswl=1.0 micron): (az, alt) in deg (93.26675455, 81.72731083)> boresight_airmass: 1.01051444909673 boresight_rotation_angle: 195.624333002684 deg boresight_rotation_coord: sky dark_time: 30.0 s datetime_end: 2022-10-05T06:53:41.357 detector_exposure_id: 40919101 detector_group: R11 detector_name: S20 detector_num: 101 detector_serial: R11_S20 exposure_id: 204595 exposure_time: 30.0 s location: (1822838.3011943279, -1062819.3254546772, -6001635.768663893) m object: unknown observation_id: 204595 observation_type: science physical_filter: i pressure: 520.0 mmHg relative_humidity: 40.0 science_program: 204595 temperature: 20.0 deg_C tracking_radec: <SkyCoord (ICRS): (ra, dec) in deg (55.67759886, -30.44239357)> visit_id: 204595 (this requires both the astro_metadata_translator and obs_lsst branches to be set up). Thanks.
            Hide
            krughoff Simon Krughoff added a comment -

            I've reviewed my pieces, I believe. Correct me if I'm wrong. It looks very good to me. Minor comments on the PR.

            Show
            krughoff Simon Krughoff added a comment - I've reviewed my pieces, I believe. Correct me if I'm wrong. It looks very good to me. Minor comments on the PR.
            Hide
            mfisherlevine Merlin Fisher-Levine added a comment -

            I'm also done with the review I think, great stuff

            Show
            mfisherlevine Merlin Fisher-Levine added a comment - I'm also done with the review I think, great stuff
            Hide
            tjenness Tim Jenness added a comment -

            Thanks for the reviews. Merged astro_metadata_translator and obs_lsst branches. We decided against the obs_base change.

            Show
            tjenness Tim Jenness added a comment - Thanks for the reviews. Merged astro_metadata_translator and obs_lsst branches. We decided against the obs_base change.

              People

              • Assignee:
                tjenness Tim Jenness
                Reporter:
                tjenness Tim Jenness
                Reviewers:
                James Chiang, Merlin Fisher-Levine, Simon Krughoff
                Watchers:
                James Chiang, Jim Bosch, Kian-Tat Lim, Merlin Fisher-Levine, Michelle Gower, Simon Krughoff, Tim Jenness
              • Votes:
                0 Vote for this issue
                Watchers:
                7 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Summary Panel