Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-33942

Northern Spring 2022 Butler schema migration

    XMLWordPrintable

    Details

    • Story Points:
      15
    • Team:
      Architecture
    • Urgent?:
      No

      Description

      This ticket is for tracking the upcoming butler schema migration tickets once the RFC is approved.

      Commissioning driven

      • To support the new definition of visit: (DM-30948)
        • Add seq_start and seq_end entries to exposure.
        • Add seq_num to the visit record (this would be the seq_start field from the exposure record).
      • Add azimuth to the exposure record. The zenith angle is already present but the azimuth was not. (DM-33859)
      • Add a visit_system to the instrument record to indicate the default that should be used if there is an ambiguity (a visit without a visit system). (DM-33818)
      • Add an entry in the exposure record ("is_simulated") to indicate that this exposure was not really on sky at all. Data are being taken in the lab with real instruments but with simulated telescope settings making it look as if the data are on sky. (DM-33728)

      Other dimension changes

      • Add healpix dimension support (DM-33946)

        Attachments

          Issue Links

            Activity

            Hide
            jbosch Jim Bosch added a comment -

            Removing myself from reviewers. I had a couple of minor comments on the obs_base and obs_lsst PRs, and of course the big one is export/import migration in daf_butler - but I think we should probably (mostly) punt on that on this ticket, and just limit ourselves to some combination of:

            • handling just a few special cases in a not-so-principled way to avoid breakage for CI-tested git-committed export files;
            • just putting the right versions into export files so we can recognize problems going forward.
            Show
            jbosch Jim Bosch added a comment - Removing myself from reviewers. I had a couple of minor comments on the obs_base and obs_lsst PRs, and of course the big one is export/import migration in daf_butler - but I think we should probably (mostly) punt on that on this ticket, and just limit ourselves to some combination of: handling just a few special cases in a not-so-principled way to avoid breakage for CI-tested git-committed export files; just putting the right versions into export files so we can recognize problems going forward.
            Hide
            mfisherlevine Merlin Fisher-Levine added a comment -

            Sorry that took me a little while to get to. I can't remember all the things that have been mentioned for the dimensions.yaml part of the review, but I do trust that they were captured on the various tickets, so as long as they're all in there, I'm confident this should be everything needed. I left a couple of minor comments on the translation code.

            Show
            mfisherlevine Merlin Fisher-Levine added a comment - Sorry that took me a little while to get to. I can't remember all the things that have been mentioned for the dimensions.yaml part of the review, but I do trust that they were captured on the various tickets, so as long as they're all in there, I'm confident this should be everything needed. I left a couple of minor comments on the translation code.
            Hide
            mfisherlevine Merlin Fisher-Levine added a comment -

            Actually, one of the big things that is very much needed is DM-30948, and which I thought was coming through in this schema migration cycle, but looks like it's not.

            The lt;dr there is that visitIds are only derivable via an awful hack, and data which goes in as an exposure and comes out as a visit cannot be retrieved expect via this hack, and that's a very bad look for users.

            Show
            mfisherlevine Merlin Fisher-Levine added a comment - Actually, one of the big things that is very much needed is DM-30948 , and which I thought was coming through in this schema migration cycle, but looks like it's not. The lt;dr there is that visitIds are only derivable via an awful hack, and data which goes in as an exposure and comes out as a visit cannot be retrieved expect via this hack, and that's a very bad look for users.
            Hide
            tjenness Tim Jenness added a comment -

            Merlin Fisher-Levine: The entire purpose of this schema change is to implement DM-30948 (see the first line of the description of this ticket). The code you reviewed on obs_lsst calculates the quantities that are needed and if you look at the obs_base PR you will see that it is all devoted to getting the new visit definition working.

            Please look at the dimensions.yaml changes on the daf_butler PR which implement all the record changes you requested.

            Show
            tjenness Tim Jenness added a comment - Merlin Fisher-Levine : The entire purpose of this schema change is to implement DM-30948 (see the first line of the description of this ticket). The code you reviewed on obs_lsst calculates the quantities that are needed and if you look at the obs_base PR you will see that it is all devoted to getting the new visit definition working. Please look at the dimensions.yaml changes on the daf_butler PR which implement all the record changes you requested.
            Hide
            mfisherlevine Merlin Fisher-Levine added a comment -

            OK, great. I couldn't remember/didn't know exactly what was required to do that, and just saw that that ticket was still To Do and figured I should mention, but glad it's all covered. (I had read all the bits you said, like I said, I just wasn't conscious of what was required for that other ticket)

            Show
            mfisherlevine Merlin Fisher-Levine added a comment - OK, great. I couldn't remember/didn't know exactly what was required to do that, and just saw that that ticket was still To Do and figured I should mention, but glad it's all covered. (I had read all the bits you said, like I said, I just wasn't conscious of what was required for that other ticket)

              People

              Assignee:
              tjenness Tim Jenness
              Reporter:
              tjenness Tim Jenness
              Reviewers:
              Merlin Fisher-Levine
              Watchers:
              Andy Salnikov, Jim Bosch, Merlin Fisher-Levine, Tim Jenness, Yusra AlSayyad
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Jenkins

                  No builds found.