Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-27858

Validate new Gaia AstrometryTask defaults for HSC

    XMLWordPrintable

Details

    • Story
    • Status: To Do
    • Resolution: Unresolved
    • None
    • None
    • No

    Description

      We need to define some verification/validation criteria for changing HSC to use Gaia DR2 for single frame astrometry (accepted in RFC-697, implementation ticket: DM-27013). This ticket is to do that HSC validation, now that Gaia is the astrometry default for everything other than HSC.

      We may want other tickets for people to develop the necessary validation criteria.

      Attachments

        Issue Links

          Activity

            No builds found.
            Parejkoj John Parejko created issue -
            Parejkoj John Parejko made changes -
            Field Original Value New Value
            Link This issue has to be finished together with DM-27013 [ DM-27013 ]
            sullivan Ian Sullivan made changes -
            Labels SciencePipelines
            Parejkoj John Parejko made changes -
            Link This issue is triggered by DM-27013 [ DM-27013 ]
            Parejkoj John Parejko made changes -
            Link This issue has to be finished together with DM-27013 [ DM-27013 ]
            Parejkoj John Parejko made changes -
            Summary Validate new Gaia AstrometryTask defaults Validate new Gaia AstrometryTask defaults for HSC
            Parejkoj John Parejko made changes -
            Summary Validate new Gaia AstrometryTask defaults for HSC Validate new Gaia AstrometryTask defaults for HSC
            Parejkoj John Parejko made changes -
            Description We need to define some verification/validation criteria for changing to use Gaia DR2 for single frame astrometry (accepted in RFC-697, implementation ticket: DM-27013). This ticket is to do that validation, once there is a PR ready for DM-27013 that can be tested on.

            We may want other tickets for people to develop the necessary validation criteria.
            We need to define some verification/validation criteria for changing HSC to use Gaia DR2 for single frame astrometry (accepted in RFC-697, implementation ticket: DM-27013). This ticket is to do that HSC validation, now that Gaia is the astrometry default for everything other than HSC.

            We may want other tickets for people to develop the necessary validation criteria.
            Parejkoj John Parejko made changes -
            Watchers Ian Sullivan, Jim Bosch, John Parejko, Lauren MacArthur, Meredith Rawls, Yusra AlSayyad [ Ian Sullivan, Jim Bosch, John Parejko, Lauren MacArthur, Meredith Rawls, Yusra AlSayyad ] Ian Sullivan, Jim Bosch, John Parejko, Lauren MacArthur, Lee Kelvin, Meredith Rawls, Yusra AlSayyad [ Ian Sullivan, Jim Bosch, John Parejko, Lauren MacArthur, Lee Kelvin, Meredith Rawls, Yusra AlSayyad ]
            Parejkoj John Parejko added a comment -

            The relevant obs_subaru changes are on the tickets/DM-27013 branch of that package (PR closed without merging), and I believe I took care of all the ancillary changes (e.g. ci_hsc_gen3) to make it useable.

            Parejkoj John Parejko added a comment - The relevant obs_subaru changes are on the tickets/ DM-27013 branch of that package ( PR closed without merging), and I believe I took care of all the ancillary changes (e.g. ci_hsc_gen3) to make it useable.

            We've been doing some validation on the new difference imaging with some HSC data, and when switching to use GAIA-DR2 as the astrometry ref cat, I'm seeing about a 20% failure rate in the calibrate step. For this particular work it's CCDs 49, 50, 57, 58, 65, and 66, so we shouldn't be suffering from edge effects much, if at all. I'll put a few examples of the job output from failures here:

            /project/kherner/DM-35283_diffim_refactor_fakes/submit/u/kherner/DM-35283/newDiffim_bestThirdSeeing_autoModeWithFakes/20220726T202826Z/jobs/calibrate/23704/d8c8e045-5ba6-4372-b75d-f767085ee4a0_calibrate_23704_66.21181331.err 

            /project/kherner/DM-35283_diffim_refactor_fakes/submit/u/kherner/DM-35283/newDiffim_bestThirdSeeing_autoModeWithFakes/20220726T202826Z/jobs/calibrate/1214/95c35431-d23e-4adb-8435-302ea7c1dc44_calibrate_1214_66.21181335.err 

            The full config settings for this run are available here:

            /repo/main/u/kherner/DM-35283/newDiffim_bestThirdSeeing_autoModeWithFakes/20220726T202826Z/calibrate_config/calibrate_config_u_kherner_DM-35283_newDiffim_bestThirdSeeing_autoModeWithFakes_20220726T202826Z.py 

            I will note that all of the failures appears to be in r-band, but I don't know if that was just luck or not.

             

            kherner Kenneth Herner added a comment - We've been doing some validation on the new difference imaging with some HSC data, and when switching to use GAIA-DR2 as the astrometry ref cat, I'm seeing about a 20% failure rate in the calibrate step. For this particular work it's CCDs 49, 50, 57, 58, 65, and 66, so we shouldn't be suffering from edge effects much, if at all. I'll put a few examples of the job output from failures here: /project/kherner/ DM-35283 _diffim_refactor_fakes/submit/u/kherner/ DM-35283 /newDiffim_bestThirdSeeing_autoModeWithFakes/20220726T202826Z/jobs/calibrate/23704/d8c8e045-5ba6-4372-b75d-f767085ee4a0_calibrate_23704_66.21181331.err  /project/kherner/ DM-35283 _diffim_refactor_fakes/submit/u/kherner/ DM-35283 /newDiffim_bestThirdSeeing_autoModeWithFakes/20220726T202826Z/jobs/calibrate/1214/95c35431-d23e-4adb-8435-302ea7c1dc44_calibrate_1214_66.21181335.err  The full config settings for this run are available here: /repo/main/u/kherner/ DM-35283 /newDiffim_bestThirdSeeing_autoModeWithFakes/20220726T202826Z/calibrate_config/calibrate_config_u_kherner_ DM-35283 _newDiffim_bestThirdSeeing_autoModeWithFakes_20220726T202826Z.py  I will note that all of the failures appears to be in r-band, but I don't know if that was just luck or not.  
            kherner Kenneth Herner added a comment - - edited

            We also tried adding the following config options to calibrate:

            config.astrometry.referenceSelector.doMagLimit = True
             config.astrometry.referenceSelector.magLimit.minimum = 16.0
             config.astrometry.referenceSelector.magLimit.fluxField = 'phot_g_mean_flux'

            We tested it on one of the failed r-band quanta (visit 1208 and detector 66 for reference), and saw no change. Just for completeness, we also learned that this was getting added to the config (must be a default somewhere?):

            config.astromRefObjLoader.anyFilterMapsToThis='phot_g_mean'

            That probably doesn't cause an issue though.

            kherner Kenneth Herner added a comment - - edited We also tried adding the following config options to calibrate: config.astrometry.referenceSelector.doMagLimit = True config.astrometry.referenceSelector.magLimit.minimum = 16.0 config.astrometry.referenceSelector.magLimit.fluxField = 'phot_g_mean_flux' We tested it on one of the failed r-band quanta (visit 1208 and detector 66 for reference), and saw no change. Just for completeness, we also learned that this was getting added to the config (must be a default somewhere?): config.astromRefObjLoader.anyFilterMapsToThis='phot_g_mean' That probably doesn't cause an issue though.
            lauren Lauren MacArthur added a comment - - edited

            TL;DR Bottom Line: if the magnitude ranges of the reference and source catalogs that get sent to the matcher are really different (in any direction), the matching will have a high likelihood of getting locked onto a terrible match and the ensuing astrometric fit will be terrible.

            Adding comments from a (lengthy!) Slack thread:
             
            Ok, I think I’ve confirmed my suspicion about the issue with the mag ranges of the source vs. ref catalogs.  Here we have the opposite situation as was noted in DM-32128 in that we have a source catalog that goes much fainter than the reference (even though this is the higher detection threshold icSrc catalog!) and the fit is getting lost. I’ve only tried on two of your problematic detectors so far, but, e.g. using the following pipeline (note the high minSnr set for the source selection):

            description: testing with gaia for SFM astrometry
            imports:
              - location: $DRP_PIPE_DIR/pipelines/HSC/DRP-RC2.yaml#calibrate
            tasks:
              calibrate:
                class: lsst.pipe.tasks.calibrate.CalibrateTask
                config:
                  connections.astromRefCat: 'gaia_dr2_20200414'
                  astromRefObjLoader.ref_dataset_name: 'gaia_dr2_20200414'
                  astromRefObjLoader.anyFilterMapsToThis: 'phot_g_mean'
                  astromRefObjLoader.filterMap: {}
                  python: >
                    config.astrometry.sourceSelector["matcher"].minSnr = 450
            

            if I run:

            $ pipetask --long-log run -d "instrument='HSC' AND skymap='hsc_rings_v1' AND visit IN (1208) and detector IN (66)" -b /repo/main --input HSC/runs/RC2/w_2022_28/DM-35609 --output u/lauren/gaiaTest -p /home/lauren/tickets/gaiaTest/gaiaTest.yaml#calibrate
            

            I get:

            detector: 66, visit: 1208:
            Matched and fit WCS in 3 iterations; found 17 matches with on-sky distance mean and scatter = 0.008 +- 0.005 arcsec
            

            and another example that previously “fails”:

            detector: 49, visit: 19658:
            Matched and fit WCS in 3 iterations; found 20 matches with on-sky distance mean and scatter = 0.009 +- 0.008 arcsec
            

            The bad news (that erykoff loves to point out  ) is that trimming the source catalog to better match the mag range of the reference catalog will be finicky at best (and certainly won’t be as simple as a one-size-fits-all S/N threshold!)

            I have some hopes that we can at least alleviate the situation to some extent with the "rough" zeropoint estimates for a given camera as the fluxMag0T1 config in isr (e.g. see HSC values).  I created DM-30993 to investigate this possibility, but...

            Some direct quotes about the difficulties from erykoff include:

            FYI, doing mag matching for culling is much, much, much more difficult when comparing to Gaia DR2 with the broad G band which is a huge mismatch to (say) z, y bands.

            And the color terms are very large even in r-band.  For bluer stars (g-i < 1.5) the r-G is well behaved, but redder than that the color term goes up to more than a magnitude at g-i~2.5.

            There’s a challenge of doing mag cuts on the source cat, because we don’t know the throughput until we can do some sort of calibration which requires matching, and … well, it’s kind of circular isn’t it?  There’s a test to do this within the iteration but you have to get that first match roughly correct or else you’re sunk.

            This [fluxMag0T1] is some estimate of the zeropoint (in flux units) for a 1s exposure, I believe.  If everything is working right.  This will probably get you within a mag or so?  The transparency can vary, the airmass can vary, and the system throughput degrades with time between mirror coatings (every 5 years or so).  So all that together you can lose a magnitude easy.

            On the slightly brighter side, he also notes:

            Using Gaia for LSST should be easier than HSC because (a) bigger detectors, and (b) shorter exposures.

            lauren Lauren MacArthur added a comment - - edited TL;DR Bottom Line: if the magnitude ranges of the reference and source catalogs that get sent to the matcher are really different (in any direction), the matching will have a high likelihood of getting locked onto a terrible match and the ensuing astrometric fit will be terrible. Adding comments from a (lengthy!) Slack thread :   Ok, I think I’ve confirmed my suspicion about the issue with the mag ranges of the source vs. ref catalogs.  Here we have the opposite situation as was noted in DM-32128 in that we have a source catalog that goes much fainter than the reference (even though this is the higher detection threshold  icSrc  catalog!) and the fit is getting lost. I’ve only tried on two of your problematic detectors so far, but, e.g. using the following pipeline (note the high minSnr set for the source selection): description: testing with gaia for SFM astrometry imports: - location: $DRP_PIPE_DIR / pipelines / HSC / DRP - RC2.yaml #calibrate tasks: calibrate: class : lsst.pipe.tasks.calibrate.CalibrateTask config: connections.astromRefCat: 'gaia_dr2_20200414' astromRefObjLoader.ref_dataset_name: 'gaia_dr2_20200414' astromRefObjLoader.anyFilterMapsToThis: 'phot_g_mean' astromRefObjLoader.filterMap: {} python: > config.astrometry.sourceSelector[ "matcher" ].minSnr = 450 if I run: $ pipetask - - long - log run - d "instrument='HSC' AND skymap='hsc_rings_v1' AND visit IN (1208) and detector IN (66)" - b / repo / main - - input HSC / runs / RC2 / w_2022_28 / DM - 35609 - - output u / lauren / gaiaTest - p / home / lauren / tickets / gaiaTest / gaiaTest.yaml #calibrate I get: detector: 66 , visit: 1208 : Matched and fit WCS in 3 iterations; found 17 matches with on - sky distance mean and scatter = 0.008 + - 0.005 arcsec and another example that previously “fails”: detector: 49 , visit: 19658 : Matched and fit WCS in 3 iterations; found 20 matches with on - sky distance mean and scatter = 0.009 + - 0.008 arcsec The bad news (that erykoff  loves to point out  ) is that trimming the source catalog to better match the mag range of the reference catalog will be finicky at best (and certainly won’t be as simple as a one-size-fits-all S/N threshold!) I have some hopes that we can at least alleviate the situation to some extent with the "rough" zeropoint estimates for a given camera as the fluxMag0T1  config in isr (e.g. see  HSC values ).  I created  DM-30993 to investigate this possibility, but... Some direct quotes about the difficulties from erykoff include: FYI, doing mag matching for culling is much, much, much more difficult when comparing to Gaia DR2 with the broad G band which is a huge mismatch to (say) z, y bands. And the color terms are very large even in r-band.  For bluer stars (g-i < 1.5) the r-G is well behaved, but redder than that the color term goes up to more than a magnitude at g-i~2.5. There’s a challenge of doing mag cuts on the source cat, because we don’t know the throughput until we can do some sort of calibration which requires matching, and … well, it’s kind of circular isn’t it?  There’s a test to do this within the iteration but you have to get that first match roughly correct or else you’re sunk. This [fluxMag0T1] is some estimate of the zeropoint (in flux units) for a 1s exposure, I believe.  If everything is working right.  This will probably get you within a mag or so?  The transparency can vary, the airmass can vary, and the system throughput degrades with time between mirror coatings (every 5 years or so).  So all that together you can lose a magnitude easy. On the slightly brighter side, he also notes: Using Gaia for LSST should be easier than HSC because (a) bigger detectors, and (b) shorter exposures.
            kherner Kenneth Herner made changes -
            Link This issue relates to DM-35283 [ DM-35283 ]
            Parejkoj John Parejko made changes -
            Link This issue relates to DM-35492 [ DM-35492 ]
            Parejkoj John Parejko made changes -
            Link This issue relates to DM-19951 [ DM-19951 ]

            I've linked DM-35492 and DM-19951 here, as I I think we might be helped quite a bit by finally adopting the FitAffineWcsTask, using the known camera geometry and orientation. Particularly for HSC, where we have a good distortion model already (whereas for DECam we don't).

            Gaia does have the bp/rp bands (and they are available in the dr2 refcat I made), which would be better for doing culling on that initial match, but they're also noisier. I don't recall offhand if we have a way to do band-dependent source or refcat culling in AstrometryTask.

            If HSC's potentially long exposure times mean that using Gaia for HSC is too much trouble, we can leave the HSC PS1 overrides in place and close this as "won't fix". Gaia's less important for HSC anyway, since it's in the north and goes so much deeper. We have LSST-appropriate defaults in place now, per DM-27013.

            Parejkoj John Parejko added a comment - I've linked DM-35492 and DM-19951 here, as I I think we might be helped quite a bit by finally adopting the FitAffineWcsTask , using the known camera geometry and orientation. Particularly for HSC, where we have a good distortion model already (whereas for DECam we don't). Gaia does have the bp/rp bands (and they are available in the dr2 refcat I made), which would be better for doing culling on that initial match, but they're also noisier. I don't recall offhand if we have a way to do band-dependent source or refcat culling in AstrometryTask. If HSC's potentially long exposure times mean that using Gaia for HSC is too much trouble, we can leave the HSC PS1 overrides in place and close this as "won't fix". Gaia's less important for HSC anyway, since it's in the north and goes so much deeper. We have LSST-appropriate defaults in place now, per DM-27013 .
            ebellm Eric Bellm made changes -
            Remote Link This issue links to "Page (Confluence)" [ 34470 ]
            Parejkoj John Parejko made changes -
            Link This issue relates to DM-36576 [ DM-36576 ]
            Parejkoj John Parejko made changes -
            Link This issue relates to DM-39836 [ DM-39836 ]

            People

              Unassigned Unassigned
              Parejkoj John Parejko
              Ian Sullivan, Jim Bosch, John Parejko, Kenneth Herner, Lauren MacArthur, Lee Kelvin, Meredith Rawls, Yusra AlSayyad
              Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

                Created:
                Updated:

                Jenkins

                  No builds found.