Analysis on DM-29819 revealed some residual issues related to loading of reference catalogs. Specifically, because of an inconsistent setting of the config parameter:
calibrate.astromRefObjLoader.pixelMargin
and the
computeVisitRegions["single-raw-wcs"].padding
of defineVisits.py
there are cases where slightly different reference catalogs are getting loaded in gen2 vs. gen3, I think the logic should be that pixelMargin <= computeVisitRegions["single-raw-wcs"].padding. However, as it stands, the default for pixelMargin is 300 and, in cases we were unlucky not to encounter in ci_hsc, this can lead to a smaller loaded region in gen3 because of the smaller visit padded definition (i.e. if a shard edge lies close to the padded visit edge). See DM-29819 for an example.

While I'm not sure it will be possible to check the above condition in a config "validation", I think we should at least ensure it is satisfied for the stack defaults as well as any overrides in individual obs_ packages.