Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-30927

Set calexp WCS to None for failed astrometric fit

    XMLWordPrintable

    Details

    • Story Points:
      42
    • Epic Link:
    • Sprint:
      DRP S21b, DRP S22A, DRP S22B, DRP F22A
    • Team:
      Data Release Production
    • Urgent?:
      No

      Description

      (Stealing language from the related DM-30921)  As per our discussion at the June 28th Metrics Meeting in order to more gently handle known types of irrecoverable single-frame processing astrometry failures, we should set the WCS of any detector for which the SFM astrometric fit failed to None and allow processing to continue. These will now warn instead of Raising.

        Attachments

          Issue Links

            Activity

            Hide
            lauren Lauren MacArthur added a comment -

            Alrighty, I think this is finally ready for you to start having a look.  Note that all my branches are circa w_2022_30.  I realize I already have some rebasing to do, but wanted to give you the same branches as were run for my RC2 DRP battle testing with that weekly.  There are at least two things that will get impacted on rebase.  One is just that all the gen2 adaptions will just go away (with all the exorcising you and Tim have been up to lately!), so feel free to skip over those.  The other is the turning on of the new image differencing task that I believe is now on main.  I will test that by rerunning the appropriate steps based off my u/lauren/DM-30927_bps collection.  If you'd prefer to review from rebased-to-current-main branches, let me know!

            Oh, and my step4 is making good progress.  All of the following had 100% success rate:

            • getTemplate 
            • forcedPhotCcd
            • writeRecalibratedSourceTable

            imageDifference had 30 failures which all look to be the same as what Brock sees (e.g. on DM-35609). The rest are chugging along...

            Show
            lauren Lauren MacArthur added a comment - Alrighty, I think this is finally ready for you to start having a look.  Note that all my branches are circa w_2022_30.  I realize I already have some rebasing to do, but wanted to give you the same branches as were run for my RC2 DRP battle testing with that weekly.  There are at least two things that will get impacted on rebase.  One is just that all the gen2 adaptions will just go away (with all the exorcising you and Tim have been up to lately!), so feel free to skip over those.  The other is the turning on of the new image differencing task that I believe is now on main.  I will test that by rerunning the appropriate steps based off my u/lauren/ DM-30927 _bps collection.  If you'd prefer to review from rebased-to-current-main branches, let me know! Oh, and my step4 is making good progress.  All of the following had 100% success rate: getTemplate  forcedPhotCcd writeRecalibratedSourceTable imageDifference had 30 failures which all look to be the same as what Brock sees (e.g. on DM-35609 ). The rest are chugging along...
            Hide
            lauren Lauren MacArthur added a comment -

            Eli Rykoff pointed out that one of my updates in makeWarp.py in pipe_tasks slightly changed the API of the prepareCalibratedExposures() function, so any current callers might be caught off-guard (a search of all of GitHub had no results, so it's very unlikely anyone will ever notice this!) It was also noted that this function really shouldn't appear public and should not be called from "the outside" anyhow. As such, the original function call signature has been annotated to be deprecated (removal after v25) and now delegates to the leading underscore version, which is now the version called in the code.

            I checked running makeWarp while still calling the deprecated version with:

            $ pipetask --long-log run -d "instrument='HSC' AND skymap='discrete/ci_hsc' AND tract=0 AND patch=69 AND band='r'" -b /sdf/data/rubin/u/laurenma/LSST/ci_hsc_gen3/DATA --input HSC/runs/ci_hsc --output u/lauren/DM-30927 -p $DRP_PIPE_DIR/pipelines/HSC/DRP-ci_hsc.yaml#makeWarp
            

            I do indeed get the following warning:

            WARNING 2022-09-20T12:13:06.419-07:00 py.warnings (makeWarp:{instrument: 'HSC', skymap: 'discrete/ci_hsc', tract: 0, patch: 69, visit: 903342, ...})(warnings.py:109) - /sdf/data/rubin/u/laurenma/LSST/pipe_tasks/python/lsst/pipe/tasks/makeWarp.py:320: FutureWarning: Call to deprecated method prepareCalibratedExposures. (This method is deprecated in favor of its leading underscore version, _prepareCalibratedfExposures().  Will be removed after v25.) -- Deprecated since version v25.0. 
              completeIndices = self.prepareCalibratedExposures(
            

            and the code proceeds to build the warp just fine.

            Show
            lauren Lauren MacArthur added a comment - Eli Rykoff pointed out that one of my updates in makeWarp.py in pipe_tasks slightly changed the API of the prepareCalibratedExposures() function, so any current callers might be caught off-guard (a search of all of GitHub had no results, so it's very unlikely anyone will ever notice this!) It was also noted that this function really shouldn't appear public and should not be called from "the outside" anyhow. As such, the original function call signature has been annotated to be deprecated (removal after v25) and now delegates to the leading underscore version, which is now the version called in the code. I checked running makeWarp while still calling the deprecated version with: $ pipetask - - long - log run - d "instrument='HSC' AND skymap='discrete/ci_hsc' AND tract=0 AND patch=69 AND band='r'" - b / sdf / data / rubin / u / laurenma / LSST / ci_hsc_gen3 / DATA - - input HSC / runs / ci_hsc - - output u / lauren / DM - 30927 - p $DRP_PIPE_DIR / pipelines / HSC / DRP - ci_hsc.yaml #makeWarp I do indeed get the following warning: WARNING 2022 - 09 - 20T12 : 13 : 06.419 - 07 : 00 py.warnings (makeWarp:{instrument: 'HSC' , skymap: 'discrete/ci_hsc' , tract: 0 , patch: 69 , visit: 903342 , ...})(warnings.py: 109 ) - / sdf / data / rubin / u / laurenma / LSST / pipe_tasks / python / lsst / pipe / tasks / makeWarp.py: 320 : FutureWarning: Call to deprecated method prepareCalibratedExposures. (This method is deprecated in favor of its leading underscore version, _prepareCalibratedfExposures(). Will be removed after v25.) - - Deprecated since version v25. 0. completeIndices = self .prepareCalibratedExposures( and the code proceeds to build the warp just fine.
            Hide
            lauren Lauren MacArthur added a comment -

            Final successful jenkins run.

            Show
            lauren Lauren MacArthur added a comment - Final successful jenkins run .
            Hide
            lauren Lauren MacArthur added a comment -

            This has been merged, but is still "off" by default (i.e. by leaving requireAstrometry=True in pipe_tasks). Another round of battle testing on a full DRP of RC2 will happen on another ticket (DM-36415: to be done as soon as such a run is feasible using bps/parsl at the USDF) and decisions on turning it on (by default or in obs_* overrides) will be considered subsequently.

            Show
            lauren Lauren MacArthur added a comment - This has been merged, but is still "off" by default (i.e. by leaving requireAstrometry=True  in  pipe_tasks ). Another round of battle testing on a full DRP of RC2 will happen on another ticket ( DM-36415 : to be done as soon as such a run is feasible using bps/parsl at the USDF) and decisions on turning it on (by default or in obs_* overrides) will be considered subsequently.
            Hide
            Parejkoj John Parejko added a comment -

            Per the config organization described in RFC-775, we don't want this in obs package overrides. I suspect it should be set in DRP specifically.

            Show
            Parejkoj John Parejko added a comment - Per the config organization described in RFC-775 , we don't want this in obs package overrides. I suspect it should be set in DRP specifically.

              People

              Assignee:
              lauren Lauren MacArthur
              Reporter:
              lauren Lauren MacArthur
              Reviewers:
              Eli Rykoff
              Watchers:
              Eli Rykoff, Jim Bosch, John Parejko, Krzysztof Findeisen, Lauren MacArthur, Yusra AlSayyad
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Jenkins Builds

                  No builds found.