Details
-
Type:
Story
-
Status: Done
-
Resolution: Done
-
Fix Version/s: None
-
Component/s: obs_subaru
-
Labels:
-
Story Points:8
-
Epic Link:
-
Sprint:DRP S21b
-
Team:Data Release Production
-
Urgent?:No
Description
Analysis on DM-29819 revealed some residual issues related to loading of reference catalogs. Specifically, because of an inconsistent setting of the config parameter:
calibrate.astromRefObjLoader.pixelMargin
and the
computeVisitRegions["single-raw-wcs"].padding
of defineVisits.py
there are cases where slightly different reference catalogs are getting loaded in gen2 vs. gen3, I think the logic should be that pixelMargin <= computeVisitRegions["single-raw-wcs"].padding. However, as it stands, the default for pixelMargin is 300 and, in cases we were unlucky not to encounter in ci_hsc, this can lead to a smaller loaded region in gen3 because of the smaller visit padded definition (i.e. if a shard edge lies close to the padded visit edge). See DM-29819 for an example.
While I'm not sure it will be possible to check the above condition in a config "validation", I think we should at least ensure it is satisfied for the stack defaults as well as any overrides in individual obs_ packages.
Attachments
Issue Links
- relates to
-
DM-30296 ap_verify HSC Gen 3 ingestion crashes on missing defineVisits config
- Done
-
- Done
-
- Done
-
- Done
- mentioned in
-
Page Loading...
Activity
Thanks for the review, Jim. I made the changes you requested (removed above noted obs_subaru config, so there's a new PR here, and added a comment to the test override in ap_association). I also did a bit more testing with other datasets just to make sure things look ok (thanks to Meredith Rawls for getting my going with DECam processing in gen3!)
The following plots show:
- full reference catalog (i.e. all of the shards associated with the visit/detector) as blue circles
- new "pre-filtered" catalog added in this ticket as orange crosses
- sources actually used in the SFM astrometry fit (astrom_used) as purple stars
- several WCS/bboxes: black = raw WCS, dotted pink = black + original 300 padding, dash-dot lime = dotted pink + 100 pixel "extra" outerRegion padding, dashed teal = calexp WCS.
In the following series of plots for the three datasets, the first plot shows the above for the original padding of 300 and the second is for the new 250 pixel padding (the pink and lime bboxes still reflect the 300, so you can see the pre-filtered cat doesn't extend all the way out to them).
HSC:
300 pixel padding
250 pixel padding
DECam:
300 pixel padding
250 pixel padding
DC2:
300 pixel padding
250 pixel padding
They all look like the ended up with identical astrometry fits...except for DC2 example, where there seems to be a dearth of astrom_used points in the upper right corner of the original 300 pixel padding, but a bunch do show up in the 250 pixel padding. I actually looked at ~twenty different detectors for this DC2 visit and this was the only one showing any difference. Given that the calexp WCSs look identical and that the 250 padding seems to be "better", I'm not concerned about this difference.
I had to do some rebasing and there's the new obs_subaru PR, so I kicked off another Jenkins (but seeing the recent traffic on #dm-build-problems, I won't be surprised if the MacOS build fails...)
Allrighty...we finally got a green Jenkins. Merged and done! Community post announcing the config default changes and potential edge cases with current gen3 repo visit definitions to follow soon.
Unfortunately, this ticket broke ap_verify as it did not remove the reference to defineVisits.py in https://github.com/lsst/obs_subaru/blob/master/config/datasetIngest-gen3.py#L7
Aargh...sorry about that. I can get in a fix this afternoon.
Looks good, but would you mind removing the obs_subaru defineVisits.py config override as well, now that it's unneccessary? (https://github.com/lsst/obs_subaru/blob/master/config/defineVisits.py)