Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-30701

Propagate astrometry offset stats in AstrometryTask and store in ExposureSummaryStats/VisitSummary

    XMLWordPrintable

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: meas_astrom, pipe_tasks
    • Labels:
      None

      Description

      Currently, the quality of the initial astrometry solution from the AstrometryTask (in terms of offset and error in the offset) is only logged and is not persisted in any way. We have found that these statistics are very useful for QA and possibly exposure selection. Therefore, we should store these in the ExposureSummaryStats attached to each calexp and bring them into the VisitSummary.

      In working on this ticket, I realized that the current computation of ExposureSummaryStats is in characterize and not calibrate, and therefore wasn't using the correct photometric calibration or wcs for computing the summary statistics. So this ticket moves the running of the task to the correct location.

        Attachments

          Issue Links

            Activity

            Hide
            erykoff Eli Rykoff added a comment -

            PRs are here: https://github.com/lsst/afw/pull/596 https://github.com/lsst/meas_astrom/pull/145 https://github.com/lsst/pipe_tasks/pull/535

            I have also run ci_hsc_gen3 locally to confirm that the offset values are getting propagated correctly.

            Show
            erykoff Eli Rykoff added a comment - PRs are here: https://github.com/lsst/afw/pull/596 https://github.com/lsst/meas_astrom/pull/145 https://github.com/lsst/pipe_tasks/pull/535 I have also run ci_hsc_gen3 locally to confirm that the offset values are getting propagated correctly.
            Hide
            lauren Lauren MacArthur added a comment - - edited

            Thanks for this...will be most useful!  I went ahead and ran processCcd on a single visit/ccd of HSC data on these branches to see how this looks.  Indeed, when I read in and look at the newly made calexp, printing calexp.getInfo().getSummaryStats() shows all the right numbers   However, when I first tried to load the calexp, I inadvertently used a shell that did not have your branches setup, so I got the following error:

            /software/lsstsw/stack_20210520/stack/miniconda3-py38_4.9.2-0.6.0/Linux64/afw/21.0.0-39-g68de21fec+f88c352dc9/python/lsst/afw/image/_exposureSummaryStats.py in _read(bytes)
                 87     @staticmethod
                 88     def _read(bytes):
            ---> 89         return ExposureSummaryStats(**yaml.load(bytes, Loader=yaml.SafeLoader))
             
            TypeError: __init__() got an unexpected keyword argument 'astromOffsetMean'
            

            So it seems this change does introduce some compatibility issues (in that newly processed calexp's can't be read in with older stacks). I do realize these situations can often be unavoidable, but I am also not entirely sure how we handle this (seems at least an announcement on community and a slack channel or two would be warranted?)  I thought you may have already hashed this out on DM-27896, but I couldn't see any commentary on the ticket, and I actually think it didn't have such issues as the calexp.getInfo().getSummaryStats() just gets set to None on reading older calexp images. Does this mean we will face this issue if/when-ever we add elements to the ExposrueSummaryStats()

            Show
            lauren Lauren MacArthur added a comment - - edited Thanks for this...will be most useful!  I went ahead and ran processCcd on a single visit/ccd of HSC data on these branches to see how this looks.  Indeed, when I read in and look at the newly made calexp , printing calexp.getInfo().getSummaryStats() shows all the right numbers   However, when I first tried to load the calexp , I inadvertently used a shell that did not have your branches setup, so I got the following error: / software / lsstsw / stack_20210520 / stack / miniconda3 - py38_4. 9.2 - 0.6 . 0 / Linux64 / afw / 21.0 . 0 - 39 - g68de21fec + f88c352dc9 / python / lsst / afw / image / _exposureSummaryStats.py in _read(bytes) 87 @ staticmethod 88 def _read(bytes): - - - > 89 return ExposureSummaryStats( * * yaml.load(bytes, Loader = yaml.SafeLoader))   TypeError: __init__() got an unexpected keyword argument 'astromOffsetMean' So it seems this change does introduce some compatibility issues (in that newly processed calexp's can't be read in with older stacks). I do realize these situations can often be unavoidable, but I am also not entirely sure how we handle this (seems at least an announcement on community and a slack channel or two would be warranted?)  I thought you may have already hashed this out on DM-27896 , but I couldn't see any commentary on the ticket, and I actually think it didn't have such issues as the calexp.getInfo().getSummaryStats() just gets set to None on reading older calexp images. Does this mean we will face this issue if/when-ever we add elements to the ExposrueSummaryStats() ? 
            Hide
            erykoff Eli Rykoff added a comment -

            Lauren MacArthur this is ready to look at again. I fixed the problem so it can't happen again on https://github.com/lsst/afw/pull/596 . That is, the ship has sailed for old stacks, but if we make further updates then it will work with a warning.

            Show
            erykoff Eli Rykoff added a comment - Lauren MacArthur this is ready to look at again. I fixed the problem so it can't happen again on https://github.com/lsst/afw/pull/596 . That is, the ship has sailed for old stacks, but if we make further updates then it will work with a warning.
            Hide
            lauren Lauren MacArthur added a comment -

            Looking at the code now...did you kick off another Jenkins?

            Show
            lauren Lauren MacArthur added a comment - Looking at the code now...did you kick off another Jenkins?
            Hide
            erykoff Eli Rykoff added a comment -
            Show
            erykoff Eli Rykoff added a comment - This was from 2 days ago and nothing has changed: https://ci.lsst.codes/blue/organizations/jenkins/stack-os-matrix/detail/stack-os-matrix/34490/pipeline
            Hide
            lauren Lauren MacArthur added a comment -

            Great, thanks.  LGTM!

            Show
            lauren Lauren MacArthur added a comment - Great, thanks.  LGTM!

              People

              Assignee:
              erykoff Eli Rykoff
              Reporter:
              erykoff Eli Rykoff
              Reviewers:
              Lauren MacArthur
              Watchers:
              Eli Rykoff, Lauren MacArthur
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Jenkins Builds

                  No builds found.