Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-20289

Make an attempt at fixing the plate scale issue in obs_subaru

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: obs_subaru
    • Labels:
      None
    • Story Points:
      12
    • Epic Link:
    • Sprint:
      Per-CCD Sprint 1
    • Team:
      Data Release Production

      Description

      For the work being done in DM-19943, we need to have the CCD plate scale description in the camera geometry to properly reflect the pixel-to-mm conversion. Currently (for historical reasons?), the obs_subaru package defines this transformation to be unity and this is a showstopper for using HSC data as a test data for validation of the work of DM-19943. While all cameras are in the process of being converted to YamlCamera and will use consistent/correct scaling (this is DM-19352 for obs_subaru), we would like to be able to move forward with this before that is all merged (HSC is otherwise the ideal test dataset for these purposes because we understand it well and the distortion model is well represented). This ticket is thus to make an attempt at fixing this units issue in the current state of the camera representation (slight emphasis on "attempt" as this might be more complicated/convoluted than one might naively picture and this should not turn into a huge time sink).

        Attachments

          Issue Links

            Activity

            Hide
            lauren Lauren MacArthur added a comment -

            Thanks Paul.  Any thoughts (wearing your NAOJ hat) on making sure meas_mosaic also gets properly converted?

            Show
            lauren Lauren MacArthur added a comment - Thanks Paul.  Any thoughts (wearing your NAOJ hat) on making sure meas_mosaic also gets properly converted?
            Hide
            price Paul Price added a comment -

            Ack, that was meant to be a private comment, but I guess I don't mind if everyone sees it!

            We don't want to decommission meas_mosaic without first discussing with Japan, but my impression is that since we've been leaning on jointcal for the latest production run, meas_mosaic will no longer be needed. However, I would feel better about it if jointcal was much faster.

            Show
            price Paul Price added a comment - Ack, that was meant to be a private comment, but I guess I don't mind if everyone sees it! We don't want to decommission meas_mosaic without first discussing with Japan, but my impression is that since we've been leaning on jointcal for the latest production run, meas_mosaic will no longer be needed. However, I would feel better about it if jointcal was much faster.
            Hide
            lauren Lauren MacArthur added a comment -

            Ok, besides meas_mosaic, I think all issues with the move to define the focal plane camera geometry of HSC in units of mm have been resolved.  The major changes were as following:

            • updating camera.py to use mm units in the top level and transform configs as well as for each individual detector
            • scaling of the distortion model coefficients (which were derived in the context of "unity" scaling) to that of mm. This is done in makeAstPolyMapCoeffs() in hoc's transformRegistry.py
            • update all config overrides that were using "unity" units (includes isr.py, vignette.py, focalBackground.py, and skyCorr.py)

            The following packages also needed updating to adapt to the new units:
            pipe_drivers, fgcmcal, and meas_mosaic (the latter just to get the tests passing). In the case of pipe_drivers, it had attempted to account for focal planes being defined in different units for different cameras, but it was missing one scaling that set an important threshold. With the fix here, the derived {{skyCorr}}s are now identical for the old vs. new unit systems.

            There are tickets/DM-20289 branches that are compatible with the tickets/DM-20154 branches. None of the tickets/DM-20289 will be merged (see below), but they may be useful for any further work on DM-20154 with HSC data. As such, the work for this ticket is effectively completed.

            The actual switch to the new mm unit focal plane scale for HSC is going to happen on a separate ticket (DM-20548) that will be isolated from the work on DM-20154 (and will have to go through an RFC process first). Further battle testing will also be done on at least one full tract from the RC2 dataset (i.e. running through singleFrameDriver, jointcal, coaddDriver, and multibandDriver).

            Show
            lauren Lauren MacArthur added a comment - Ok, besides meas_mosaic , I think all issues with the move to define the focal plane camera geometry of HSC in units of mm have been resolved.  The major changes were as following: updating  camera.py to use mm units in the top level and transform configs as well as for each individual detector scaling of the distortion model coefficients (which were derived in the context of "unity" scaling) to that of mm. This is done in makeAstPolyMapCoeffs() in hoc's transformRegistry.py update all config overrides that were using "unity" units (includes isr.py , vignette.py , focalBackground.py , and skyCorr.py ) The following packages also needed updating to adapt to the new units: pipe_drivers , fgcmcal , and meas_mosaic (the latter just to get the tests passing). In the case of pipe_drivers , it had attempted to account for focal planes being defined in different units for different cameras, but it was missing one scaling that set an important threshold. With the fix here, the derived {{skyCorr}}s are now identical for the old vs. new unit systems. There are tickets/ DM-20289 branches that are compatible with the tickets/ DM-20154 branches. None of the tickets/ DM-20289 will be merged (see below), but they may be useful for any further work on DM-20154 with HSC data. As such, the work for this ticket is effectively completed. The actual switch to the new mm unit focal plane scale for HSC is going to happen on a separate ticket ( DM-20548 ) that will be isolated from the work on DM-20154 (and will have to go through an RFC process first). Further battle testing will also be done on at least one full tract from the RC2 dataset (i.e. running through singleFrameDriver , jointcal , coaddDriver , and multibandDriver ).
            Hide
            lauren Lauren MacArthur added a comment -

            Would you mind having a look and confirming if the work done here is sufficient to close this out?

            Show
            lauren Lauren MacArthur added a comment - Would you mind having a look and confirming if the work done here is sufficient to close this out?
            Hide
            lauren Lauren MacArthur added a comment - - edited

            Oh, and a final Jenkins with:

            PRODUCTS: lsst_distrib lsst_ci ci_hsc
            REFS: tickets/DM-20289 tickets/DM-20154
            

            is green.

            Show
            lauren Lauren MacArthur added a comment - - edited Oh, and a final Jenkins with: PRODUCTS: lsst_distrib lsst_ci ci_hsc REFS: tickets/DM-20289 tickets/DM-20154 is green .

              People

              • Assignee:
                lauren Lauren MacArthur
                Reporter:
                lauren Lauren MacArthur
                Reviewers:
                Jim Bosch
                Watchers:
                Chris Morrison, Eli Rykoff, Jim Bosch, John Parejko, John Swinbank, Lauren MacArthur, Paul Price
              • Votes:
                0 Vote for this issue
                Watchers:
                7 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Summary Panel