Uploaded image for project: 'Request For Comments'
  1. Request For Comments
  2. RFC-194

Names of rotation columns in Exposure and Visit tables in baseline schema

    XMLWordPrintable

    Details

    • Type: RFC
    • Status: Implemented
    • Resolution: Done
    • Component/s: DM
    • Labels:
      None

      Description

      DM-1785 is supposed to add new column to Visit and Exposure tables in baseline schema, this column will contain sky rotation angle. While working on it I noticed that schema contains rotation column which is camera rotation angle (in alt/az frame). To avoid possible confusion I propose to rename existing column to make it more explicit. Additional concern for existing column is that documentation does not specify whether camera angle (and altitude/azimuth) is determined from hardware, either expectation or measure, or does it come from later analysis, so we are also seeking input on what documentation for those three columns should say.

      Current column definition (see also here):

          rotation DOUBLE NOT NULL,
              -- <descr>Rotation of the camera.</descr>
              -- <unit>deg</unit>
      

      With this proposal it is going to change to (description for altitude/azimuth will be updated as well):

          cameraRotation DOUBLE NOT NULL,
              -- <descr>Rotation angle of the camera in alt-azimuthal frame as determined from <source>?.</descr>
              -- <unit>deg</unit>
      

      The new column for sky rotation angle will be defined as:

          skyRotation DOUBLE NOT NULL,
              -- <descr>Sky rotation angle.</descr>
              -- <unit>deg</unit>
      

      Please comment on the names of the columns and preferred source of the camera rotation angle. I'll be on vacation for a big part of the next seven days but will try to follow up.

        Attachments

          Issue Links

            Activity

            Hide
            ktl Kian-Tat Lim added a comment - - edited

            @rhl There are many things in the baseline schema for which the source is currently unclear. Would it be a good idea to draw up a chart of differences between what is expected to be produced (by which pipeline and in which data product) and what is in the schema? (This should be easier when LDM-151 has been revised.)

            Show
            ktl Kian-Tat Lim added a comment - - edited @rhl There are many things in the baseline schema for which the source is currently unclear. Would it be a good idea to draw up a chart of differences between what is expected to be produced (by which pipeline and in which data product) and what is in the schema? (This should be easier when LDM-151 has been revised.)
            Hide
            rhl Robert Lupton added a comment -

            I think it would. If only because we could clarify exactly what is being loaded, and ensure that it's actually available.

            Show
            rhl Robert Lupton added a comment - I think it would. If only because we could clarify exactly what is being loaded, and ensure that it's actually available.
            Hide
            gpdf Gregory Dubois-Felsmann added a comment - - edited

            Regarding John Parejko's question "Why is this needed separately from the WCS?" I think I would ask in return that we first figure out "What are the visit and image metadata tables for?" Are they just for end users, mainly to help them identify the images that exist? Are they also for the production system, to help organize the work to be performed in (e.g.) a data release? (These questions help us understand whether the ra, dec, rotation, etc., should be TCS-level data, results of Level 1, or results of our best data-release-grade astrometric calibration, and whether in fact there may be multiple tables representing the development of this information as processing proceeds - I am assuming we wish to avoid refinement-in-place.) Are they used both for finding images and for "I have an image (in memory? as an exported file?) and now I want to find out more about the conditions that produced it"?

            Is the image metadata table supposed to allow me to answer a query like "find all the images that contain this (ra, dec)"? How precise should the answer be if based just on the table? Does it take gaps in the focal plane into account? Fiducial regions of the CCDs away from difficult edge defects? Can it take the final WCS and astrometric calibration into account? (How can it, if the WCS for a single image - see RFC-193 - is so complex that it can only be represented by binary tables / data blobs?) Probably it should not and need not attempt to be so precise, IMO.

            Lots of questions, no answers, but a larger issue arises:

            How do we make decisions of this nature? I don't mean a process answer like "the TCT has the authority", but more like: who should be reasoning about things like this and recommending a coherent set of answers for a larger group to comment/vote on? I'm nervous about these questions being answered by default and piecemeal by the individual developers who are implementing the creation of various data items. In DM-1785, Kian-Tat Lim mentioned "the collection of DM scientists" in this context.

            I very much the idea of drawing up an explicit spec for the source of each data item mentioned in the schema.


            A separate point: indeed all of the information like set points, read-backs, etc. should be in the Engineering and Facilities Database. However, that database, per se, is organized purely temporally as a record of telemetry. That is, it will say things like "at time X (which we are hoping is a TAI time) the mount was commanded to slew to (ra,dec)", "at time Y the mount reported completing a slew, reporting (ra2,dec2,alt,az, etc.), and commenced sidereal tracking", "at time Z after the shutter opened, the guider system reported the pointing to be (ra3,dec3, etc.)". Each channel of telemetry will have its own temporally organized table. By correlating image acquisition times and shutter motion times with the the telemetry time stamps, it's possible to determine the telemetry values that are relevant to a particular image. But the EFD will not itself contain the "materialized view" (of a sort) that this represents. It is stated to be a DM job to build that view.

            This is a moderately complicated job, because some telemetry channels will have one measurement per image, some one per visit, some will have many per image (e.g., FPA temperatures read out at 1 Hz), and some perhaps will have rates less than the imaging cadence and so will require interpolation from outside the image integration time window in order to construct the view. We hope to build a small number of standard "strategies" for this and make an appropriate choice of strategy for each channel.

            A possible answer to Simon Krughoff's concerns about redundant and conflicting information is that perhaps in "the" Exposure metadata table as conceived in the present schema we would choose to limit the information to a narrow and non-redundant set – but that then there will also be other tables indexed by image ID or visit ID in which the additional information is available.

            In other words, the current schema is a somewhat idealized or naive view of what will really be in the DM databases.

            Show
            gpdf Gregory Dubois-Felsmann added a comment - - edited Regarding John Parejko 's question "Why is this needed separately from the WCS?" I think I would ask in return that we first figure out "What are the visit and image metadata tables for?" Are they just for end users, mainly to help them identify the images that exist? Are they also for the production system, to help organize the work to be performed in (e.g.) a data release? (These questions help us understand whether the ra, dec, rotation, etc., should be TCS-level data, results of Level 1, or results of our best data-release-grade astrometric calibration, and whether in fact there may be multiple tables representing the development of this information as processing proceeds - I am assuming we wish to avoid refinement-in-place.) Are they used both for finding images and for "I have an image (in memory? as an exported file?) and now I want to find out more about the conditions that produced it"? Is the image metadata table supposed to allow me to answer a query like "find all the images that contain this (ra, dec)"? How precise should the answer be if based just on the table? Does it take gaps in the focal plane into account? Fiducial regions of the CCDs away from difficult edge defects? Can it take the final WCS and astrometric calibration into account? (How can it, if the WCS for a single image - see RFC-193 - is so complex that it can only be represented by binary tables / data blobs?) Probably it should not and need not attempt to be so precise, IMO. Lots of questions, no answers, but a larger issue arises: How do we make decisions of this nature? I don't mean a process answer like "the TCT has the authority", but more like: who should be reasoning about things like this and recommending a coherent set of answers for a larger group to comment/vote on? I'm nervous about these questions being answered by default and piecemeal by the individual developers who are implementing the creation of various data items. In DM-1785 , Kian-Tat Lim mentioned "the collection of DM scientists" in this context. I very much the idea of drawing up an explicit spec for the source of each data item mentioned in the schema. A separate point: indeed all of the information like set points, read-backs, etc. should be in the Engineering and Facilities Database. However, that database, per se , is organized purely temporally as a record of telemetry. That is, it will say things like "at time X (which we are hoping is a TAI time) the mount was commanded to slew to (ra,dec)", "at time Y the mount reported completing a slew, reporting (ra2,dec2,alt,az, etc. ), and commenced sidereal tracking", "at time Z after the shutter opened, the guider system reported the pointing to be (ra3,dec3, etc. )". Each channel of telemetry will have its own temporally organized table. By correlating image acquisition times and shutter motion times with the the telemetry time stamps, it's possible to determine the telemetry values that are relevant to a particular image. But the EFD will not itself contain the "materialized view" (of a sort) that this represents. It is stated to be a DM job to build that view. This is a moderately complicated job, because some telemetry channels will have one measurement per image, some one per visit, some will have many per image (e.g., FPA temperatures read out at 1 Hz), and some perhaps will have rates less than the imaging cadence and so will require interpolation from outside the image integration time window in order to construct the view. We hope to build a small number of standard "strategies" for this and make an appropriate choice of strategy for each channel. A possible answer to Simon Krughoff 's concerns about redundant and conflicting information is that perhaps in " the " Exposure metadata table as conceived in the present schema we would choose to limit the information to a narrow and non-redundant set – but that then there will also be other tables indexed by image ID or visit ID in which the additional information is available. In other words, the current schema is a somewhat idealized or naive view of what will really be in the DM databases.
            Hide
            salnikov Andy Salnikov added a comment -

            I'm back from my trip. Reading through the comments it looks like the sensible thing to do for now is what Simon proposed - add skyRotation column and remove cameraRotation until we know exactly what we need. Unless there are objections to this I'm going to implement this change to schema tomorrow.

            Show
            salnikov Andy Salnikov added a comment - I'm back from my trip. Reading through the comments it looks like the sensible thing to do for now is what Simon proposed - add skyRotation column and remove cameraRotation until we know exactly what we need. Unless there are objections to this I'm going to implement this change to schema tomorrow.
            Hide
            salnikov Andy Salnikov added a comment -

            No new comments received, adopting.

            Show
            salnikov Andy Salnikov added a comment - No new comments received, adopting.

              People

              Assignee:
              salnikov Andy Salnikov
              Reporter:
              salnikov Andy Salnikov
              Watchers:
              Andy Salnikov, David Ciardi [X] (Inactive), Gregory Dubois-Felsmann, Hsin-Fang Chiang, Jim Bosch, John Parejko, Kian-Tat Lim, Mario Juric, Robert Lupton, Simon Krughoff, Tim Jenness, Zeljko Ivezic
              Votes:
              0 Vote for this issue
              Watchers:
              12 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:
                Planned End:

                  Jenkins Builds

                  No builds found.