# Validate new Gaia AstrometryTask defaults for HSC

XMLWordPrintable

#### Details

• Type: Story
• Status: To Do
• Resolution: Unresolved
• Fix Version/s: None
• Component/s: None
• Labels:
• Urgent?:
No

#### Description

We need to define some verification/validation criteria for changing HSC to use Gaia DR2 for single frame astrometry (accepted in RFC-697, implementation ticket: DM-27013). This ticket is to do that HSC validation, now that Gaia is the astrometry default for everything other than HSC.

We may want other tickets for people to develop the necessary validation criteria.

#### Activity

Hide
John Parejko added a comment -

The relevant obs_subaru changes are on the tickets/DM-27013 branch of that package (PR closed without merging), and I believe I took care of all the ancillary changes (e.g. ci_hsc_gen3) to make it useable.

Show
John Parejko added a comment - The relevant obs_subaru changes are on the tickets/ DM-27013 branch of that package ( PR closed without merging), and I believe I took care of all the ancillary changes (e.g. ci_hsc_gen3) to make it useable.
Hide
Kenneth Herner added a comment -

We've been doing some validation on the new difference imaging with some HSC data, and when switching to use GAIA-DR2 as the astrometry ref cat, I'm seeing about a 20% failure rate in the calibrate step. For this particular work it's CCDs 49, 50, 57, 58, 65, and 66, so we shouldn't be suffering from edge effects much, if at all. I'll put a few examples of the job output from failures here:

/project/kherner/DM-35283_diffim_refactor_fakes/submit/u/kherner/DM-35283/newDiffim_bestThirdSeeing_autoModeWithFakes/20220726T202826Z/jobs/calibrate/23704/d8c8e045-5ba6-4372-b75d-f767085ee4a0_calibrate_23704_66.21181331.err

The full config settings for this run are available here:

/repo/main/u/kherner/DM-35283/newDiffim_bestThirdSeeing_autoModeWithFakes/20220726T202826Z/calibrate_config/calibrate_config_u_kherner_DM-35283_newDiffim_bestThirdSeeing_autoModeWithFakes_20220726T202826Z.py

I will note that all of the failures appears to be in r-band, but I don't know if that was just luck or not.

Show
Kenneth Herner added a comment - We've been doing some validation on the new difference imaging with some HSC data, and when switching to use GAIA-DR2 as the astrometry ref cat, I'm seeing about a 20% failure rate in the calibrate step. For this particular work it's CCDs 49, 50, 57, 58, 65, and 66, so we shouldn't be suffering from edge effects much, if at all. I'll put a few examples of the job output from failures here: /project/kherner/ DM-35283 _diffim_refactor_fakes/submit/u/kherner/ DM-35283 /newDiffim_bestThirdSeeing_autoModeWithFakes/20220726T202826Z/jobs/calibrate/23704/d8c8e045-5ba6-4372-b75d-f767085ee4a0_calibrate_23704_66.21181331.err  /project/kherner/ DM-35283 _diffim_refactor_fakes/submit/u/kherner/ DM-35283 /newDiffim_bestThirdSeeing_autoModeWithFakes/20220726T202826Z/jobs/calibrate/1214/95c35431-d23e-4adb-8435-302ea7c1dc44_calibrate_1214_66.21181335.err  The full config settings for this run are available here: /repo/main/u/kherner/ DM-35283 /newDiffim_bestThirdSeeing_autoModeWithFakes/20220726T202826Z/calibrate_config/calibrate_config_u_kherner_ DM-35283 _newDiffim_bestThirdSeeing_autoModeWithFakes_20220726T202826Z.py  I will note that all of the failures appears to be in r-band, but I don't know if that was just luck or not.
Hide
Kenneth Herner added a comment - - edited

We also tried adding the following config options to calibrate:

 config.astrometry.referenceSelector.doMagLimit = True  config.astrometry.referenceSelector.magLimit.minimum = 16.0  config.astrometry.referenceSelector.magLimit.fluxField = 'phot_g_mean_flux'

We tested it on one of the failed r-band quanta (visit 1208 and detector 66 for reference), and saw no change. Just for completeness, we also learned that this was getting added to the config (must be a default somewhere?):

That probably doesn't cause an issue though.

Show
Kenneth Herner added a comment - - edited We also tried adding the following config options to calibrate: config.astrometry.referenceSelector.doMagLimit = True config.astrometry.referenceSelector.magLimit.minimum = 16.0 config.astrometry.referenceSelector.magLimit.fluxField = 'phot_g_mean_flux' We tested it on one of the failed r-band quanta (visit 1208 and detector 66 for reference), and saw no change. Just for completeness, we also learned that this was getting added to the config (must be a default somewhere?): config.astromRefObjLoader.anyFilterMapsToThis='phot_g_mean' That probably doesn't cause an issue though.
Hide
Lauren MacArthur added a comment - - edited

TL;DR Bottom Line: if the magnitude ranges of the reference and source catalogs that get sent to the matcher are really different (in any direction), the matching will have a high likelihood of getting locked onto a terrible match and the ensuing astrometric fit will be terrible.

Ok, I think I’ve confirmed my suspicion about the issue with the mag ranges of the source vs. ref catalogs.  Here we have the opposite situation as was noted in DM-32128 in that we have a source catalog that goes much fainter than the reference (even though this is the higher detection threshold icSrc catalog!) and the fit is getting lost. I’ve only tried on two of your problematic detectors so far, but, e.g. using the following pipeline (note the high minSnr set for the source selection):

 description: testing with gaia for SFM astrometry imports:  - location: $DRP_PIPE_DIR/pipelines/HSC/DRP-RC2.yaml#calibrate tasks:  calibrate:  class: lsst.pipe.tasks.calibrate.CalibrateTask  config:  connections.astromRefCat: 'gaia_dr2_20200414'  astromRefObjLoader.ref_dataset_name: 'gaia_dr2_20200414'  astromRefObjLoader.anyFilterMapsToThis: 'phot_g_mean'  astromRefObjLoader.filterMap: {}  python: >  config.astrometry.sourceSelector["matcher"].minSnr = 450  if I run:  $ pipetask --long-log run -d "instrument='HSC' AND skymap='hsc_rings_v1' AND visit IN (1208) and detector IN (66)" -b /repo/main --input HSC/runs/RC2/w_2022_28/DM-35609 --output u/lauren/gaiaTest -p /home/lauren/tickets/gaiaTest/gaiaTest.yaml#calibrate 

I get:

 detector: 66, visit: 1208: Matched and fit WCS in 3 iterations; found 17 matches with on-sky distance mean and scatter = 0.008 +- 0.005 arcsec 

and another example that previously “fails”:

 detector: 49, visit: 19658: Matched and fit WCS in 3 iterations; found 20 matches with on-sky distance mean and scatter = 0.009 +- 0.008 arcsec 

The bad news (that Eli Rykoff loves to point out  ) is that trimming the source catalog to better match the mag range of the reference catalog will be finicky at best (and certainly won’t be as simple as a one-size-fits-all S/N threshold!)

I have some hopes that we can at least alleviate the situation to some extent with the "rough" zeropoint estimates for a given camera as the fluxMag0T1 config in isr (e.g. see HSC values).  I created DM-30993 to investigate this possibility, but...

Some direct quotes about the difficulties from Eli Rykoff include:

FYI, doing mag matching for culling is much, much, much more difficult when comparing to Gaia DR2 with the broad G band which is a huge mismatch to (say) z, y bands.

And the color terms are very large even in r-band.  For bluer stars (g-i < 1.5) the r-G is well behaved, but redder than that the color term goes up to more than a magnitude at g-i~2.5.

There’s a challenge of doing mag cuts on the source cat, because we don’t know the throughput until we can do some sort of calibration which requires matching, and … well, it’s kind of circular isn’t it?  There’s a test to do this within the iteration but you have to get that first match roughly correct or else you’re sunk.

This [fluxMag0T1] is some estimate of the zeropoint (in flux units) for a 1s exposure, I believe.  If everything is working right.  This will probably get you within a mag or so?  The transparency can vary, the airmass can vary, and the system throughput degrades with time between mirror coatings (every 5 years or so).  So all that together you can lose a magnitude easy.

On the slightly brighter side, he also notes:

Using Gaia for LSST should be easier than HSC because (a) bigger detectors, and (b) shorter exposures.

Show
Lauren MacArthur added a comment - - edited TL;DR Bottom Line: if the magnitude ranges of the reference and source catalogs that get sent to the matcher are really different (in any direction), the matching will have a high likelihood of getting locked onto a terrible match and the ensuing astrometric fit will be terrible. Adding comments from a (lengthy!) Slack thread :   Ok, I think I’ve confirmed my suspicion about the issue with the mag ranges of the source vs. ref catalogs.  Here we have the opposite situation as was noted in DM-32128 in that we have a source catalog that goes much fainter than the reference (even though this is the higher detection threshold  icSrc  catalog!) and the fit is getting lost. I’ve only tried on two of your problematic detectors so far, but, e.g. using the following pipeline (note the high minSnr set for the source selection): description: testing with gaia for SFM astrometry imports: - location: $DRP_PIPE_DIR / pipelines / HSC / DRP - RC2.yaml #calibrate tasks: calibrate: class : lsst.pipe.tasks.calibrate.CalibrateTask config: connections.astromRefCat: 'gaia_dr2_20200414' astromRefObjLoader.ref_dataset_name: 'gaia_dr2_20200414' astromRefObjLoader.anyFilterMapsToThis: 'phot_g_mean' astromRefObjLoader.filterMap: {} python: > config.astrometry.sourceSelector[ "matcher" ].minSnr = 450 if I run:$ pipetask - - long - log run - d "instrument='HSC' AND skymap='hsc_rings_v1' AND visit IN (1208) and detector IN (66)" - b / repo / main - - input HSC / runs / RC2 / w_2022_28 / DM - 35609 - - output u / lauren / gaiaTest - p / home / lauren / tickets / gaiaTest / gaiaTest.yaml #calibrate I get: detector: 66 , visit: 1208 : Matched and fit WCS in 3 iterations; found 17 matches with on - sky distance mean and scatter = 0.008 + - 0.005 arcsec and another example that previously “fails”: detector: 49 , visit: 19658 : Matched and fit WCS in 3 iterations; found 20 matches with on - sky distance mean and scatter = 0.009 + - 0.008 arcsec The bad news (that Eli Rykoff  loves to point out  ) is that trimming the source catalog to better match the mag range of the reference catalog will be finicky at best (and certainly won’t be as simple as a one-size-fits-all S/N threshold!) I have some hopes that we can at least alleviate the situation to some extent with the "rough" zeropoint estimates for a given camera as the fluxMag0T1  config in isr (e.g. see  HSC values ).  I created  DM-30993 to investigate this possibility, but... Some direct quotes about the difficulties from Eli Rykoff include: FYI, doing mag matching for culling is much, much, much more difficult when comparing to Gaia DR2 with the broad G band which is a huge mismatch to (say) z, y bands. And the color terms are very large even in r-band.  For bluer stars (g-i < 1.5) the r-G is well behaved, but redder than that the color term goes up to more than a magnitude at g-i~2.5. There’s a challenge of doing mag cuts on the source cat, because we don’t know the throughput until we can do some sort of calibration which requires matching, and … well, it’s kind of circular isn’t it?  There’s a test to do this within the iteration but you have to get that first match roughly correct or else you’re sunk. This [fluxMag0T1] is some estimate of the zeropoint (in flux units) for a 1s exposure, I believe.  If everything is working right.  This will probably get you within a mag or so?  The transparency can vary, the airmass can vary, and the system throughput degrades with time between mirror coatings (every 5 years or so).  So all that together you can lose a magnitude easy. On the slightly brighter side, he also notes: Using Gaia for LSST should be easier than HSC because (a) bigger detectors, and (b) shorter exposures.
Hide
John Parejko added a comment -

I've linked DM-35492 and DM-19951 here, as I I think we might be helped quite a bit by finally adopting the FitAffineWcsTask, using the known camera geometry and orientation. Particularly for HSC, where we have a good distortion model already (whereas for DECam we don't).

Gaia does have the bp/rp bands (and they are available in the dr2 refcat I made), which would be better for doing culling on that initial match, but they're also noisier. I don't recall offhand if we have a way to do band-dependent source or refcat culling in AstrometryTask.

If HSC's potentially long exposure times mean that using Gaia for HSC is too much trouble, we can leave the HSC PS1 overrides in place and close this as "won't fix". Gaia's less important for HSC anyway, since it's in the north and goes so much deeper. We have LSST-appropriate defaults in place now, per DM-27013.

Show
John Parejko added a comment - I've linked DM-35492 and DM-19951 here, as I I think we might be helped quite a bit by finally adopting the FitAffineWcsTask , using the known camera geometry and orientation. Particularly for HSC, where we have a good distortion model already (whereas for DECam we don't). Gaia does have the bp/rp bands (and they are available in the dr2 refcat I made), which would be better for doing culling on that initial match, but they're also noisier. I don't recall offhand if we have a way to do band-dependent source or refcat culling in AstrometryTask. If HSC's potentially long exposure times mean that using Gaia for HSC is too much trouble, we can leave the HSC PS1 overrides in place and close this as "won't fix". Gaia's less important for HSC anyway, since it's in the north and goes so much deeper. We have LSST-appropriate defaults in place now, per DM-27013 .

#### People

Assignee:
Unassigned
Reporter:
John Parejko
Watchers:
Ian Sullivan, Jim Bosch, John Parejko, Kenneth Herner, Lauren MacArthur, Lee Kelvin, Meredith Rawls, Yusra AlSayyad