Uploaded image for project: 'Request For Comments'
  1. Request For Comments
  2. RFC-836

Schema changes for butler

    XMLWordPrintable

    Details

    • Type: RFC
    • Status: Implemented
    • Resolution: Done
    • Component/s: DM
    • Labels:
      None

      Description

      A number of schema changes are required to support the new way that the commissioning team wish to define visits. Given that any schema changes are disruptive we would like to make a number of changes all at once.

      Motivated by Commissioning

      • Visits are now defined from the first sequence number in the visit and the expected end sequence number. Visits can now only have one or two snaps.
      • The exposure record therefore needs seq_start and seq_end fields and the visit record needs to have seq_num added since the visit is now defined in terms of the first sequence number in the visit. We also wish to store the group_id in the visit record to allow groups of visits to be found.
      • Some commissioning tasks require the azimuth to be recorded for the exposure. Zenith angle is already recorded. The average azimuth will also be added to the visit record.
      • Visit systems are now explicitly either always one exposure per visit or else one or two exposures per visit based on the seq_start/seq_end exposure values. There is no longer the concept of arbitrary visit systems as originally envisaged. Given that in both systems the visit_id will now be identical for the first exposure in the visit, we wish to register a default visit system per-instrument that can be used to deal with an ambiguous dataId.
      • We are now taking data at the summit which look like on-sky observations even though the instrument is not on the telescope. In order to make it possible to reliably filter out these observations from real on-sky observations we wish to add an "is_simulated" boolean field to the exposure record. This will be true if any system involved in the observation is simulated.

      Other dimension changes

      • Add support for healpix dimensions.

      Internal registry changes

      • Remove ON DELETE CASCADE from collection tag rows. Currently a dataset can be deleted from a RUN even if it is part of a TAGGED collection. This is considered dangerous since you may not be aware that it was in a TAGGED collection.

      Notes

      • Adding amplifier dimensions (DM-29584) is not being considered for this RFC. Is what we currently have usable by the commissioning team?
      • Opaque dataIds are something we need but feel that the implementation is sufficiently under-specified (DM-33751) that we should defer the work to a later schema change.

        Attachments

          Issue Links

            Activity

            tjenness Tim Jenness created issue -
            tjenness Tim Jenness made changes -
            Field Original Value New Value
            Link This issue is triggering DM-33942 [ DM-33942 ]
            tjenness Tim Jenness made changes -
            Description A number of schema changes are required to support the new way that the commissioning team wish to define visits. Given that any schema changes are disruptive we would like to make a number of changes all at once.

            h2. Motivated by Commissioning

            * Visits are now defined from the first sequence number in the visit and the expected end sequence number. Visits can now only have one or two snaps.
            * The {{exposure}} record therefore needs {{seq_start}} and {{seq_end}} fields and the {{visit}} record needs to have {{seq_num}} added since the visit is now defined in terms of the first sequence number in the visit.
            * Some commissioning tasks require the azimuth to be recorded for the exposure. Zenith angle is already recorded. The average azimuth will also be added to the visit record.
            * Visit systems are now explicitly either always one exposure per visit or else one or two exposures per visit based on the seq_start/seq_end exposure values. There is no longer the concept of arbitrary visit systems as originally envisaged. Given that in both systems the visit_id will now be identical for the first exposure in the visit, we wish to register a default visit system per-instrument that can be used to deal with an ambiguous dataId.
            * We are now taking data at the summit which look like on-sky observations even though the instrument is not on the telescope. In order to make it possible to reliably filter out these observations from real on-sky observations we wish to add an "is_simulated" boolean field to the exposure record. This will be true if any system involved in the observation is simulated.

            h2. Other dimension changes

            * Add support for healpix dimensions.

            h2. Internal registry changes

            * Remove ON DELETE CASCADE from collection tag rows. Currently a dataset can be deleted from a RUN even if it is part of a TAGGED collection. This is considered dangerous since you may not be aware that it was in a TAGGED collection.

            h2. Notes

            * Adding amplifier dimensions (DM-29584) is not being considered for this RFC. Is what we currently have usable by the commissioning team?
            * Opaque dataIds are something we need but feel that the implementation is sufficiently under-specified (DM-33751) that we should defer the work to a later schema change.
            A number of schema changes are required to support the new way that the commissioning team wish to define visits. Given that any schema changes are disruptive we would like to make a number of changes all at once.

            h2. Motivated by Commissioning

            * Visits are now defined from the first sequence number in the visit and the expected end sequence number. Visits can now only have one or two snaps.
            * The {{exposure}} record therefore needs {{seq_start}} and {{seq_end}} fields and the {{visit}} record needs to have {{seq_num}} added since the visit is now defined in terms of the first sequence number in the visit. We also wish to store the group_id in the visit record to allow groups of visits to be found.
            * Some commissioning tasks require the azimuth to be recorded for the exposure. Zenith angle is already recorded. The average azimuth will also be added to the visit record.
            * Visit systems are now explicitly either always one exposure per visit or else one or two exposures per visit based on the seq_start/seq_end exposure values. There is no longer the concept of arbitrary visit systems as originally envisaged. Given that in both systems the visit_id will now be identical for the first exposure in the visit, we wish to register a default visit system per-instrument that can be used to deal with an ambiguous dataId.
            * We are now taking data at the summit which look like on-sky observations even though the instrument is not on the telescope. In order to make it possible to reliably filter out these observations from real on-sky observations we wish to add an "is_simulated" boolean field to the exposure record. This will be true if any system involved in the observation is simulated.

            h2. Other dimension changes

            * Add support for healpix dimensions.

            h2. Internal registry changes

            * Remove ON DELETE CASCADE from collection tag rows. Currently a dataset can be deleted from a RUN even if it is part of a TAGGED collection. This is considered dangerous since you may not be aware that it was in a TAGGED collection.

            h2. Notes

            * Adding amplifier dimensions (DM-29584) is not being considered for this RFC. Is what we currently have usable by the commissioning team?
            * Opaque dataIds are something we need but feel that the implementation is sufficiently under-specified (DM-33751) that we should defer the work to a later schema change.
            tjenness Tim Jenness made changes -
            Remote Link This issue links to "Page (Confluence)" [ 32407 ]
            lguy Leanne Guy made changes -
            Status Proposed [ 10805 ] Flagged [ 10606 ]
            lguy Leanne Guy made changes -
            Status Flagged [ 10606 ] Board Recommended [ 11405 ]
            tjenness Tim Jenness made changes -
            Status Board Recommended [ 11405 ] Adopted [ 10806 ]
            tjenness Tim Jenness made changes -
            Remote Link This issue links to "Page (Confluence)" [ 32478 ]
            tjenness Tim Jenness made changes -
            Resolution Done [ 10000 ]
            Status Adopted [ 10806 ] Implemented [ 11105 ]
            tjenness Tim Jenness made changes -
            Remote Link This issue links to "Page (Confluence)" [ 33617 ]

              People

              Assignee:
              tjenness Tim Jenness
              Reporter:
              tjenness Tim Jenness
              Watchers:
              Andy Salnikov, Colin Slater, Eli Rykoff, Eric Bellm, Gregory Dubois-Felsmann, Jim Bosch, John Parejko, Kian-Tat Lim, Leanne Guy, Merlin Fisher-Levine, Michelle Butler [X] (Inactive), Peter Ferguson, Robert Lupton, Steve Pietrowicz, Tim Jenness, Wil O'Mullane, Yusra AlSayyad
              Votes:
              0 Vote for this issue
              Watchers:
              17 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:
                Planned End:

                  Jenkins

                  No builds found.