I have updated the PR and will be running jenkins shortly (both ci_imsim and ci_hsc_gen3 are passing locally).
I initially just wanted to make minimal changes to allow faro to pass on both, but it was getting difficult to follow the logic of the more complex run methods so I made some steps towards
DM-31061 by changing those with catalogs, photoCalibs, astromCalibs arguments to instead use a Dict[str, List[CalibratedCatalog], where CalibratedCatalog is a frozen dataclass with a SourceCatalog and optional calibs. My logic for this is that as far as I can tell, all of the subsequent methods using these lists of catalogs and calibs expect them all to be the same length and in the same order (ie for zipping), so it makes more sense to have one single object storing related data.
I did add some much-needed type hints where I could but did not go further in implementing the use of dicts of CalibratedCatalog across faro for a couple of reasons:
1. Once faro starts using object tables instead of source catalogs (
DM-31459, DM-31460) most of this will be a moot point. This is virtually a requirement for some tasks like wPerp to run at all. Although I 'fixed' it so that it can run locally, wPerp currently uses over 100GB of memory to merge catalogs, which is exactly what objectTable is doing anyway. So I don't want to put a lot of effort into refactoring code that's just going to be wiped soon anyway. You may want to reconsider the priority of DM-31061 in light of this.
2. I'm not sure if you actually want to homogenize on a Dict[str, List] everywhere (where str is a band). You might want to consider attaching the dataId to the CalibratedCatalog (or, in the future, object table) instead. But either way the run methods need some kind of validation or separation between those that expect exactly one catalog vs multiple, one band vs multiple, etc.