Another snag shielded from the limited ci_hsc datasets. There are cases in RC2 where slightly different reference catalogs are getting loaded in gen2 vs. gen3, leading to slight differences in the WCS solutions (which then percolates...) This is related to the issue of
DM-28936 which got us most of the way there, but I now see that the change of the visit padding (from 4000 to 250 pixels) on DM-24024 should have been accompanied by an associated change in the pixelMargin config associated with the LoadReferenceObjectsConfig. I think the logic should be that pixelMargin <= computeVisitRegions["single-raw-wcs"].padding. However, as it stands, the default for pixelMargin is 300 and, in cases we were unlucky not to encounter in ci_hsc, this can lead to a smaller loaded region in gen3 because of the smaller visit padded definition (i.e. if a shard edge lies close to the padded visit edge). As an example, here are the trimmed reference catalogs for gen2 vs. gen3:
and this includes the full loaded catalogs (note that the blue gen3 shard doesn't quite cover all of the red x's of the filtered gen2 calatog, whose full loaded catalog (purple) is very differently distributed to the gen3 version...):
I have created
DM-30030 to put in a fix for this.