Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-30943

Turn on delta mag rejection for astrometry matching in configs for LSSTCam-imSim

    XMLWordPrintable

    Details

    • Story Points:
      1
    • Epic Link:
    • Sprint:
      DRP S21b
    • Team:
      Data Release Production
    • Urgent?:
      No

      Description

      There has been much recent attention drawn to a certain mode of bad astrometric fits in SFM in the context of DC2/imsim data processing (see DM-30466 for discussion and diagnosis).  A potential "fix" was added on DM-30490 in the form of an optional step in SFM astrometric fitting to perform a rejection on delta mag (ref - src) outliers in the match catalog before sending it to the fitter.  Fairly extensive testing has proven this to be a most desirable option in that it fixes (i.e. a good astrometric fit is now found) almost all of the bad fit cases (the pesky "hot mess" of DM-30466 is the only exception we have noted thus far) and it does-no-harm for the cases that were not affected and already had good fits (see DM-30812 for details of this validation).  As such, this option should be turned on by default for LSSTCam-imSim processing.  The config parameter to be added is:

      config.astrometry.doMagnitudeOutlierRejection=True

        Attachments

          Issue Links

          relates to

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-30466 Investigate bad SFM WCS fits in DP0.1 data

          • Done

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-30490 Add delta-magnitude check to AstrometryTask

          • Done

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-30812 Compare the data products of the gen2 vs. gen3 w_2021_24 DC2 runs up to Single Frame Processing

          • Done
          mentioned in

          Page Loading...

          Page Loading...

            Activity

            Ascending order - Click to sort in descending order
            Hide
            Permalink
            lauren Lauren MacArthur added a comment -

            To test this, I ran one of our bad fit examples on master and having set up this ticket branch:

            $ pipetask run -b /repo/dc2 -i 2.2i/runs/DP0.1,2.2i/calib -o u/lauren/test -p $OBS_LSST_DIR/pipelines/imsim/DRP.yaml#processCcd -d "instrument='LSSTCam-imSim' AND exposure=193111 AND detector=155"

            Master:

            calibrate.astrometry.referenceSelector INFO: Selected 272/1545 references
            		 
            calibrate.astrometry.matcher INFO: Matched 69 sources
            		 
            calibrate.astrometry.matcher INFO: Matched 70 sources
            		 
            calibrate.astrometry.matcher INFO: Matched 61 sources
            		 
            calibrate.astrometry INFO: Matched and fit WCS in 3 iterations; found 61 matches with scatter = 1.235 +- 1.014 arcsec

            This branch:

            calibrate.astrometry.matcher INFO: Matched 69 sources
            		 
            calibrate.astrometry INFO: Rough zeropoint from astrometry matches is 32.1774 +/- 0.0191.
            		 
            calibrate.astrometry INFO: Removed 12 magnitude outliers out of 69 total astrometry matches.
            		 
            calibrate.astrometry INFO: Fit WCS iter 2 failed; using previous iteration: Unable to match sources
            		 
            calibrate.astrometry INFO: Matched and fit WCS in 1 iterations; found 57 matches with scatter = 0.007 +- 0.004 arcsec

            So all looks good!

            Show
            lauren Lauren MacArthur added a comment - To test this, I ran one of our bad fit examples on master and having set up this ticket branch: $ pipetask run - b / repo / dc2 - i 2.2i / runs / DP0. 1 , 2.2i / calib - o u / lauren / test - p $OBS_LSST_DIR / pipelines / imsim / DRP.yaml #processCcd -d "instrument='LSSTCam-imSim' AND exposure=193111 AND detector=155" Master: calibrate.astrometry.referenceSelector INFO: Selected 272 / 1545 references calibrate.astrometry.matcher INFO: Matched 69 sources calibrate.astrometry.matcher INFO: Matched 70 sources calibrate.astrometry.matcher INFO: Matched 61 sources calibrate.astrometry INFO: Matched and fit WCS in 3 iterations; found 61 matches with scatter = 1.235 + - 1.014 arcsec This branch: calibrate.astrometry.matcher INFO: Matched 69 sources calibrate.astrometry INFO: Rough zeropoint from astrometry matches is 32.1774 + / - 0.0191 . calibrate.astrometry INFO: Removed 12 magnitude outliers out of 69 total astrometry matches. calibrate.astrometry INFO: Fit WCS iter 2 failed; using previous iteration: Unable to match sources calibrate.astrometry INFO: Matched and fit WCS in 1 iterations; found 57 matches with scatter = 0.007 + - 0.004 arcsec So all looks good!
            Hide
            Permalink
            lauren Lauren MacArthur added a comment - - edited

            Jenkins + ci_imsim is green.

            Show
            lauren Lauren MacArthur added a comment - - edited Jenkins + ci_imsim is green .
            Hide
            Permalink
            lauren Lauren MacArthur added a comment -

            Would you mind giving this a look? Can you think of any other check I should do before merging??

            Show
            lauren Lauren MacArthur added a comment - Would you mind giving this a look? Can you think of any other check I should do before merging??
            Hide
            Permalink
            erykoff Eli Rykoff added a comment -

            Looks good to me!

            Show
            erykoff Eli Rykoff added a comment - Looks good to me!
            Hide
            Permalink
            lauren Lauren MacArthur added a comment -

            Might go down on record for speediest review! Merged and done.

            Show
            lauren Lauren MacArthur added a comment - Might go down on record for speediest review! Merged and done.

              People

              Assignee:
              lauren Lauren MacArthur
              Reporter:
              lauren Lauren MacArthur
              Reviewers:
              Eli Rykoff
              Watchers:
              Dominique Boutigny, Eli Rykoff, James Chiang, Jim Bosch, Johann Cohen-Tanugi, Lauren MacArthur, Yusra AlSayyad
              Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Jenkins

                  No builds found.