Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-35389

Fix exposure number breaking when changing MJD

    XMLWordPrintable

    Details

    • Type: Improvement
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: ts_aos
    • Labels:

      Description

      It is possible to set `MJD` of observation date-time in policy/teleSetting.yaml . However, if one changes the value from eg. the default "59580" to "59580.15" (forward by few hours to eg. avoid twilight light with quickbackground cmd option), then  the day of observation changes from 20211231 to 20220101 , but since the visitId is hardcoded, this leads to a failure (attached error trace). error_trace_MJD.txt

      This is because when butler creates exposure and visit using define-visits , it includes, among other things, the obsDate in `YYMMDD` format, as inferred from the ingested raw data header) , but  ts_phosim  to  build a pipetask query uses a hard-coded visitId and obsId, such as

       self.visitIdOffset = 4021114100000
       obsId = 9006000
       exposure IN (\{self.visitIdOffset+obsId})
      

      So that the first exposure number is always

      4021114100000+9006000=4021123106000
      

      This is confirmed by looking at the registry of the ingested raws:

       

      butler query-data-ids . exposure visit --where "instrument='LSSTCam' "
      

      which yields 

       

      band instrument physical_filter    exposure       visit    
      ---- ---------- --------------- ------------- -------------
         g    LSSTCam               g 4022010106000 4022010106000
      

       

      This ticket is to correct that behavior so that change of MJD in `teleSetting.yaml` does not lead to a failure. 

       

        Attachments

          Issue Links

            Activity

            No builds found.
            ksuberlak Krzysztof Suberlak created issue -
            Hide
            tjenness Tim Jenness added a comment -

            For most cases the visitId will be identical to the exposure ID of the first exposure in the visit. define-visit looks solely at the exposure.id value (there are cases where it's not related to the exposure ID at all but those should be edge cases now with the new butler schema). Trying to construct your own visit ID with integer arithmetic is not going to be reliable.

            The exposure ID is calculated from the DAYOBS and SEQNUM (and CONTRLLR) headers (not explicitly the DATE-OBS header) so if your simulation is filling in DAYOBS from the MJD that is where the problem is coming from. A data query can include visit metadata directly. So you can say visit.day_obs = 20211231 and visit.seq_num = 6000, for example. This is designed to allow people to not have to ever guess what a visit ID is going to look like.

            Show
            tjenness Tim Jenness added a comment - For most cases the visitId will be identical to the exposure ID of the first exposure in the visit. define-visit looks solely at the exposure.id value (there are cases where it's not related to the exposure ID at all but those should be edge cases now with the new butler schema). Trying to construct your own visit ID with integer arithmetic is not going to be reliable. The exposure ID is calculated from the DAYOBS and SEQNUM (and CONTRLLR) headers (not explicitly the DATE-OBS header) so if your simulation is filling in DAYOBS from the MJD that is where the problem is coming from. A data query can include visit metadata directly. So you can say visit.day_obs = 20211231 and visit.seq_num = 6000 , for example. This is designed to allow people to not have to ever guess what a visit ID is going to look like.
            ksuberlak Krzysztof Suberlak made changes -
            Field Original Value New Value
            Remote Link This issue links to "Page (Confluence)" [ 33718 ]
            ksuberlak Krzysztof Suberlak made changes -
            Status To Do [ 10001 ] In Progress [ 3 ]
            ajc Andrew Connolly made changes -
            Remote Link This issue links to "Page (Confluence)" [ 33854 ]
            ksuberlak Krzysztof Suberlak made changes -
            Reviewers Bryce Kalmbach [ jbkalmbach ]
            Status In Progress [ 3 ] In Review [ 10004 ]
            jbkalmbach Bryce Kalmbach made changes -
            Status In Review [ 10004 ] Reviewed [ 10101 ]
            ksuberlak Krzysztof Suberlak made changes -
            Resolution Done [ 10000 ]
            Status Reviewed [ 10101 ] Done [ 10002 ]
            sthomas Sandrine Thomas made changes -
            Remote Link This issue links to "Page (Confluence)" [ 33969 ]
            sthomas Sandrine Thomas made changes -
            Remote Link This issue links to "Page (Confluence)" [ 33969 ]

              People

              Assignee:
              ksuberlak Krzysztof Suberlak
              Reporter:
              ksuberlak Krzysztof Suberlak
              Reviewers:
              Bryce Kalmbach
              Watchers:
              Bryce Kalmbach, Krzysztof Suberlak, Tim Jenness
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Jenkins

                  No builds found.