Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-29187

LATISS UTC/TAI problems were fixed on 2021-02-12

    XMLWordPrintable

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: obs_lsst
    • Labels:
      None
    • Story Points:
      0.5
    • Team:
      Architecture
    • Urgent?:
      No

      Description

      The obs_lsst LATISS metadata translator assumes that data after RASTART_IS_BAD is off by 37s, but in fact this was fixed on 2021-02-12. Please fix the metadata translator.

      I'd do it myself, but I don't understand the comment

      # RASTART/DECSTART/RAEND/DECEND used wrong telescope location before this
      # 2020-02-01T00:00 we fixed the telescope location, but RASTART is still
      # in mount coordinates, so off by pointing model.
      RASTART_IS_BAD = Time("2020-05-01T00:00", format="isot", scale="utc")
      

      and would prefer you to clarify that the problem being fixed is not this one.

      There is a more serious problem too, namely that the correction of c. 0.154167 is applied multiple times. The RAEND in the header is 130.211695172908, but:

      exposure.getMetadata().get("RAEND"), butler.get("raw_md", dataId).get("RAEND")
      (130.82836183957568, 130.36586183957567)
      

        Attachments

          Issue Links

            Activity

            Hide
            rhl Robert Lupton added a comment -

            I'm going to push an end date for the UTC correction to u/rhl/DM-29187 (which means that the LATISS data headers are correct), but I'm not planning to take over this ticket.

            Show
            rhl Robert Lupton added a comment - I'm going to push an end date for the UTC correction to u/rhl/ DM-29187 (which means that the LATISS data headers are correct), but I'm not planning to take over this ticket.
            Hide
            rhl Robert Lupton added a comment -

            Furthermore the RASTART test fails:

                def test_fix_header(self):
                    from astro_metadata_translator import fix_header
                    from astro_metadata_translator.tests import read_test_file
                    # Test that header fix up is working
                    # Not all headers are used in metadata translation
                    test_data = (
                        ("latiss-AT_O_20210212_000006.yaml",
                         dict(RASTART=260.1785517385836)),
                        ("latiss-AT_O_20210210_000011.yaml",
                         dict(RASTART=355.41750341182313)),
                    )
            

            It appears to use test data taken on the day that this problem was fixed!

            Show
            rhl Robert Lupton added a comment - Furthermore the RASTART test fails: def test_fix_header(self): from astro_metadata_translator import fix_header from astro_metadata_translator.tests import read_test_file # Test that header fix up is working # Not all headers are used in metadata translation test_data = ( ("latiss-AT_O_20210212_000006.yaml", dict(RASTART=260.1785517385836)), ("latiss-AT_O_20210210_000011.yaml", dict(RASTART=355.41750341182313)), ) It appears to use test data taken on the day that this problem was fixed!
            Hide
            rhl Robert Lupton added a comment -

            Oh, sorry. The how-to-repeat is

            repo = "/lsstdata/offline/teststand/NCSA_auxTel/gen2repo"
            butler = dafPersist.Butler(repo)
             
            dataId = dict(dayObs="2021-03-11" , seqNum=4)
             
            calexp = butler.get("raw", dataId)
            print(exposure.getMetadata().get("RAEND"), butler.get("raw_md", dataId).get("RAEND"))
            

            Show
            rhl Robert Lupton added a comment - Oh, sorry. The how-to-repeat is repo = "/lsstdata/offline/teststand/NCSA_auxTel/gen2repo" butler = dafPersist.Butler(repo)   dataId = dict(dayObs="2021-03-11" , seqNum=4)   calexp = butler.get("raw", dataId) print(exposure.getMetadata().get("RAEND"), butler.get("raw_md", dataId).get("RAEND"))
            Hide
            tjenness Tim Jenness added a comment -

            That RASTART_IS_BAD date is when the values were complete garbage and should not be used.

            https://github.com/lsst/obs_lsst/blob/master/python/lsst/obs/lsst/translators/latiss.py#L63 defines the RASTART_IS_HOURS date at which that was fixed to be degrees. There is no date for fixing the 37s correction because no-one had reported it fixed (see the discussion in DM-28783)

            Show
            tjenness Tim Jenness added a comment - That RASTART_IS_BAD date is when the values were complete garbage and should not be used. https://github.com/lsst/obs_lsst/blob/master/python/lsst/obs/lsst/translators/latiss.py#L63 defines the RASTART_IS_HOURS date at which that was fixed to be degrees. There is no date for fixing the 37s correction because no-one had reported it fixed (see the discussion in DM-28783 )
            Hide
            tjenness Tim Jenness added a comment -

            Do I assume that after the 12th the RASTART/DECSTART are correct in the header and do not need any fixes?

            I think for the double correction I'm going to have to finally bite the bullet and add a sentinel to the header indicating that it's been fixed. I've been wary of doing that because I didn't want to surprise people when something weird appears in their header. Maybe I should use something like HIERARCH ASTROMETADATA HEADER FIXED = TRUE.

            Show
            tjenness Tim Jenness added a comment - Do I assume that after the 12th the RASTART/DECSTART are correct in the header and do not need any fixes? I think for the double correction I'm going to have to finally bite the bullet and add a sentinel to the header indicating that it's been fixed. I've been wary of doing that because I didn't want to surprise people when something weird appears in their header. Maybe I should use something like HIERARCH ASTROMETADATA HEADER FIXED = TRUE .
            Hide
            tjenness Tim Jenness added a comment -

            To clarify the situation inherited from DM-28783:

            • At 2021-02-11T1845Z the headers were decimal degrees but had a 37s error in RASTART.
            • At 2021-02-12T0000Z the 37s was fixed.

            So it was less than 6 hours of data where the 37s error was a real problem.

            Show
            tjenness Tim Jenness added a comment - To clarify the situation inherited from DM-28783 : At 2021-02-11T1845Z the headers were decimal degrees but had a 37s error in RASTART. At 2021-02-12T0000Z the 37s was fixed. So it was less than 6 hours of data where the 37s error was a real problem.
            Hide
            tjenness Tim Jenness added a comment -

            Furthermore, the RASTART test will fail because even as I was fixing the translator to deal with the problem of the time offset the problem had been fixed but I had not been informed. This means that the data I was looking at was already correct. The reason I thought it needed fixing was that this did make the number closer to the target RA (260.178 [fixed] vs 260.181 [demand] vs 260.024 [header]).

            Show
            tjenness Tim Jenness added a comment - Furthermore, the RASTART test will fail because even as I was fixing the translator to deal with the problem of the time offset the problem had been fixed but I had not been informed. This means that the data I was looking at was already correct. The reason I thought it needed fixing was that this did make the number closer to the target RA (260.178 [fixed] vs 260.181 [demand] vs 260.024 [header] ).
            Hide
            tjenness Tim Jenness added a comment -

            I took your fix and sorted out the test.

            I'm going to do the double fixup fix in a different ticket because that can be done entirely inside astro_metadata_translator.

            Show
            tjenness Tim Jenness added a comment - I took your fix and sorted out the test. I'm going to do the double fixup fix in a different ticket because that can be done entirely inside astro_metadata_translator.
            Hide
            rhl Robert Lupton added a comment -

            Looks fine

            Show
            rhl Robert Lupton added a comment - Looks fine

              People

              Assignee:
              tjenness Tim Jenness
              Reporter:
              rhl Robert Lupton
              Reviewers:
              Robert Lupton
              Watchers:
              Michael Reuter, Robert Lupton, Tim Jenness
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Jenkins

                  No builds found.