Another snag shielded from the limited ci_hsc datasets. There are cases in RC2 where slightly different reference catalogs are getting loaded in gen2 vs. gen3, leading to slight differences in the WCS solutions (which then percolates...) This is related to the issue of DM-28936 which got us most of the way there, but I now see that the change of the visit padding (from 4000 to 250 pixels) on DM-24024 should have been accompanied by an associated change in the pixelMargin config associated with the LoadReferenceObjectsConfig. I think the logic should be that pixelMargin <= computeVisitRegions["single-raw-wcs"].padding. However, as it stands, the default for pixelMargin is 300 and, in cases we were unlucky not to encounter in ci_hsc, this can lead to a smaller loaded region in gen3 because of the smaller visit padded definition (i.e. if a shard edge lies close to the padded visit edge). As an example, here are the trimmed reference catalogs for gen2 vs. gen3:
and this includes the full loaded catalogs (note that the blue gen3 shard doesn't quite cover all of the red x's of the filtered gen2 calatog, whose full loaded catalog (purple) is very differently distributed to the gen3 version...):
I have created DM-30030 to put in a fix for this.
Doh...hit a snag with HSC-Y, so can't claim bitwise identical just yet. See
DM-29881for details (and solution, hopefully!)Comparing gen2 vs. gen3 calexp image/mask/variance arrays and photoCalib's