Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-30927

Set calexp WCS to None for failed astrometric fit

    XMLWordPrintable

Details

    • 42
    • DRP S21b, DRP S22A, DRP S22B, DRP S23A
    • Data Release Production
    • No

    Description

      (Stealing language from the related DM-30921)  As per our discussion at the June 28th Metrics Meeting in order to more gently handle known types of irrecoverable single-frame processing astrometry failures, we should set the WCS of any detector for which the SFM astrometric fit failed to None and allow processing to continue. These will now warn instead of Raising.

      Attachments

        Issue Links

          Activity

            Alrighty, I think this is finally ready for you to start having a look.  Note that all my branches are circa w_2022_30.  I realize I already have some rebasing to do, but wanted to give you the same branches as were run for my RC2 DRP battle testing with that weekly.  There are at least two things that will get impacted on rebase.  One is just that all the gen2 adaptions will just go away (with all the exorcising you and Tim have been up to lately!), so feel free to skip over those.  The other is the turning on of the new image differencing task that I believe is now on main.  I will test that by rerunning the appropriate steps based off my u/lauren/DM-30927_bps collection.  If you'd prefer to review from rebased-to-current-main branches, let me know!

            Oh, and my step4 is making good progress.  All of the following had 100% success rate:

            • getTemplate 
            • forcedPhotCcd
            • writeRecalibratedSourceTable

            imageDifference had 30 failures which all look to be the same as what Brock sees (e.g. on DM-35609). The rest are chugging along...

            lauren Lauren MacArthur added a comment - Alrighty, I think this is finally ready for you to start having a look.  Note that all my branches are circa w_2022_30.  I realize I already have some rebasing to do, but wanted to give you the same branches as were run for my RC2 DRP battle testing with that weekly.  There are at least two things that will get impacted on rebase.  One is just that all the gen2 adaptions will just go away (with all the exorcising you and Tim have been up to lately!), so feel free to skip over those.  The other is the turning on of the new image differencing task that I believe is now on main.  I will test that by rerunning the appropriate steps based off my u/lauren/ DM-30927 _bps collection.  If you'd prefer to review from rebased-to-current-main branches, let me know! Oh, and my step4 is making good progress.  All of the following had 100% success rate: getTemplate  forcedPhotCcd writeRecalibratedSourceTable imageDifference had 30 failures which all look to be the same as what Brock sees (e.g. on DM-35609 ). The rest are chugging along...

            erykoff pointed out that one of my updates in makeWarp.py in pipe_tasks slightly changed the API of the prepareCalibratedExposures() function, so any current callers might be caught off-guard (a search of all of GitHub had no results, so it's very unlikely anyone will ever notice this!) It was also noted that this function really shouldn't appear public and should not be called from "the outside" anyhow. As such, the original function call signature has been annotated to be deprecated (removal after v25) and now delegates to the leading underscore version, which is now the version called in the code.

            I checked running makeWarp while still calling the deprecated version with:

            $ pipetask --long-log run -d "instrument='HSC' AND skymap='discrete/ci_hsc' AND tract=0 AND patch=69 AND band='r'" -b /sdf/data/rubin/u/laurenma/LSST/ci_hsc_gen3/DATA --input HSC/runs/ci_hsc --output u/lauren/DM-30927 -p $DRP_PIPE_DIR/pipelines/HSC/DRP-ci_hsc.yaml#makeWarp
            

            I do indeed get the following warning:

            WARNING 2022-09-20T12:13:06.419-07:00 py.warnings (makeWarp:{instrument: 'HSC', skymap: 'discrete/ci_hsc', tract: 0, patch: 69, visit: 903342, ...})(warnings.py:109) - /sdf/data/rubin/u/laurenma/LSST/pipe_tasks/python/lsst/pipe/tasks/makeWarp.py:320: FutureWarning: Call to deprecated method prepareCalibratedExposures. (This method is deprecated in favor of its leading underscore version, _prepareCalibratedfExposures().  Will be removed after v25.) -- Deprecated since version v25.0. 
              completeIndices = self.prepareCalibratedExposures(
            

            and the code proceeds to build the warp just fine.

            lauren Lauren MacArthur added a comment - erykoff pointed out that one of my updates in makeWarp.py in pipe_tasks slightly changed the API of the prepareCalibratedExposures() function, so any current callers might be caught off-guard (a search of all of GitHub had no results, so it's very unlikely anyone will ever notice this!) It was also noted that this function really shouldn't appear public and should not be called from "the outside" anyhow. As such, the original function call signature has been annotated to be deprecated (removal after v25) and now delegates to the leading underscore version, which is now the version called in the code. I checked running makeWarp while still calling the deprecated version with: $ pipetask - - long - log run - d "instrument='HSC' AND skymap='discrete/ci_hsc' AND tract=0 AND patch=69 AND band='r'" - b / sdf / data / rubin / u / laurenma / LSST / ci_hsc_gen3 / DATA - - input HSC / runs / ci_hsc - - output u / lauren / DM - 30927 - p $DRP_PIPE_DIR / pipelines / HSC / DRP - ci_hsc.yaml #makeWarp I do indeed get the following warning: WARNING 2022 - 09 - 20T12 : 13 : 06.419 - 07 : 00 py.warnings (makeWarp:{instrument: 'HSC' , skymap: 'discrete/ci_hsc' , tract: 0 , patch: 69 , visit: 903342 , ...})(warnings.py: 109 ) - / sdf / data / rubin / u / laurenma / LSST / pipe_tasks / python / lsst / pipe / tasks / makeWarp.py: 320 : FutureWarning: Call to deprecated method prepareCalibratedExposures. (This method is deprecated in favor of its leading underscore version, _prepareCalibratedfExposures(). Will be removed after v25.) - - Deprecated since version v25. 0. completeIndices = self .prepareCalibratedExposures( and the code proceeds to build the warp just fine.

            Final successful jenkins run.

            lauren Lauren MacArthur added a comment - Final successful jenkins run .

            This has been merged, but is still "off" by default (i.e. by leaving requireAstrometry=True in pipe_tasks). Another round of battle testing on a full DRP of RC2 will happen on another ticket (DM-36415: to be done as soon as such a run is feasible using bps/parsl at the USDF) and decisions on turning it on (by default or in obs_* overrides) will be considered subsequently.

            lauren Lauren MacArthur added a comment - This has been merged, but is still "off" by default (i.e. by leaving requireAstrometry=True  in  pipe_tasks ). Another round of battle testing on a full DRP of RC2 will happen on another ticket ( DM-36415 : to be done as soon as such a run is feasible using bps/parsl at the USDF) and decisions on turning it on (by default or in obs_* overrides) will be considered subsequently.

            Per the config organization described in RFC-775, we don't want this in obs package overrides. I suspect it should be set in DRP specifically.

            Parejkoj John Parejko added a comment - Per the config organization described in RFC-775 , we don't want this in obs package overrides. I suspect it should be set in DRP specifically.

            People

              lauren Lauren MacArthur
              lauren Lauren MacArthur
              Eli Rykoff
              Eli Rykoff, Jim Bosch, John Parejko, Krzysztof Findeisen, Lauren MacArthur, Yusra AlSayyad
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Jenkins

                  No builds found.