Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-24575

Add observational and detector metadata to Registry dimension tables

    XMLWordPrintable

    Details

    • Story Points:
      4
    • Team:
      Architecture
    • Urgent?:
      No

      Description

      It's clear we're missing a lot of standard, useful columns (many of them already enumerated in the ObservationInfo class) in the exposure, visit, and detector tables. We should try harder to get this list complete before we declare the Registry schema stable to reduce the number of migrations we have to do afterwards.

      This includes:

      • Target name (for ObsCore)
      • RA/Dec/rotation angle/rotation coord (so that we can calculate visits without reading the raw data file)

      Also includes changes to obs_base to use the ra/dec in visit region calculation.

        Attachments

          Issue Links

            Activity

            Hide
            krzys Krzysztof Findeisen added a comment -

            I've updated the Gen 3 repository in ap_verify_testdata to match the new schema. That should be the only repository of ours that would affect Jenkins.

            Show
            krzys Krzysztof Findeisen added a comment - I've updated the Gen 3 repository in ap_verify_testdata to match the new schema. That should be the only repository of ours that would affect Jenkins.
            Hide
            tjenness Tim Jenness added a comment -

            No-one is asking for additional metadata so I may as well put this into review. If people want more they will have to do it on a separate ticket.

            Regarding the define-visits performance. When I started this ticket the time it took to process 8 HSC visits was 19 seconds (total, including process start up and importing python code). Now by calculating the WCS using registry and by caching in DM-26317 it's 5 seconds – the actual visit region calculations are less than a second total.

            The region calculation itself is fast enough that I now wonder if it's worth doing that as part of raw ingest (or at least having an exposure_detector_region table) – visit region then using exposure region directly. Granted calibration observations wouldn't want entries.

            Show
            tjenness Tim Jenness added a comment - No-one is asking for additional metadata so I may as well put this into review. If people want more they will have to do it on a separate ticket. Regarding the define-visits performance. When I started this ticket the time it took to process 8 HSC visits was 19 seconds (total, including process start up and importing python code). Now by calculating the WCS using registry and by caching in DM-26317 it's 5 seconds – the actual visit region calculations are less than a second total. The region calculation itself is fast enough that I now wonder if it's worth doing that as part of raw ingest (or at least having an exposure_detector_region table) – visit region then using exposure region directly. Granted calibration observations wouldn't want entries.
            Hide
            jbosch Jim Bosch added a comment - - edited

            A few comments on the PR; mostly I think we should try to add these new ones to visit as well.

            The region calculation itself is fast enough that I now wonder if it's worth doing that as part of raw ingest (or at least having an exposure_detector_region table) – visit region then using exposure region directly. Granted calibration observations wouldn't want entries.
             

            I'd like to hold off on that until we have a use case, because it complicates the query logic that picks which regions to use to relate various dimensions.  Nothing insurmountable, but a bit of a pain, at least.

            Show
            jbosch Jim Bosch added a comment - - edited A few comments on the PR; mostly I think we should try to add these new ones to visit as well. The region calculation itself is fast enough that I now wonder if it's worth doing that as part of raw ingest (or at least having an exposure_detector_region table) – visit region then using exposure region directly. Granted calibration observations wouldn't want entries.   I'd like to hold off on that until we have a use case, because it complicates the query logic that picks which regions to use to relate various dimensions.  Nothing insurmountable, but a bit of a pain, at least.
            Hide
            tjenness Tim Jenness added a comment -

            I had added target name and science program to visit (it's in the obs_base PR) but I had failed to push my daf_butler branch. Updated now.

            As for others: I can add zenith angle since that's in ObservationInfo. Is a straight average what people want to turn up in the visit table? cc/ Eli Rykoff?

            ra/dec/sky angle seem tricky as well given that they are not guaranteed to be the same for all exposures in a visit (we have dithering and no guarantee the rotator hasn't moved).

            Show
            tjenness Tim Jenness added a comment - I had added target name and science program to visit (it's in the obs_base PR) but I had failed to push my daf_butler branch. Updated now. As for others: I can add zenith angle since that's in ObservationInfo. Is a straight average what people want to turn up in the visit table? cc/ Eli Rykoff ? ra/dec/sky angle seem tricky as well given that they are not guaranteed to be the same for all exposures in a visit (we have dithering and no guarantee the rotator hasn't moved).
            Hide
            tjenness Tim Jenness added a comment -

            Merged with the zenith angle addition.

            Show
            tjenness Tim Jenness added a comment - Merged with the zenith angle addition.

              People

              Assignee:
              tjenness Tim Jenness
              Reporter:
              jbosch Jim Bosch
              Reviewers:
              Jim Bosch
              Watchers:
              Christopher Waters, Colin Slater, Gregory Dubois-Felsmann, Jim Bosch, Kenny Lo, Krzysztof Findeisen, Tim Jenness
              Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Jenkins

                  No builds found.