Details
-
Type:
Improvement
-
Status: Won't Fix
-
Resolution: Done
-
Fix Version/s: None
-
Component/s: None
-
Labels:None
-
Story Points:0
-
Epic Link:
-
Team:Data Release Production
Description
While Fred Moolekamp was chasing the errors in DM-29907, it was realized that `afwTable.SourceCatalog` forbids getting children if the catalog is unsorted. In checking if the catalog is sorted, it performs an operation that is O( n), which is as expensive as fetching the children even from an unsorted catalog. Moreover, this check is performed for each parent, thereby making getting the children for all parents an O(n^2) operation. However, it is desirable to check (somewhere, once) if the catalog is sorted, as "unsortedness" could indicate a potentially serious problem elsewhere. This ticket captures potential solutions, and discussions made towards that on Slack.
Attachments
Issue Links
Activity
Weirdly, ci_hsc_gen3 doesn't have build issues, only ci_hsc_gen2 does. Is this a concern?
This happens rarely enough that we don't have a policy (AFAIK) for whether to make it another ticket, but I would recommend another ticket just to avoid various kinds of clashes complicating things.
There are some things ci_hsc_gen2 does that ci_hsc_gen3 does not, and they take different code paths in much of the I/O. I wouldn't have expected them to differ on this ticket, but that's not a strong expectation and I wouldn't be able to say more without looking much more closely.
Alright. Marking this as Won't Fix now and will create a new PR with the patch included in DM-30105.
To conclude the gen2/gen3 differences in this ticket: in forcedPhotCcd.py, ci_hsc_gen2 gets the `refCat` by calling the `fetchReferences` method from `runDataRef` whereas, ci_hsc_gen3 gets the same by calling `mergeAndFilterReferences` method in `runQuantum`. Only the first method takes the code path that contained the bug, so it is perfectly reasonable that ci_hsc_gen3 build passed but not ci_hsc_gen2.
I think I've fixed it now. There was a logic bug in the `subset` method `references.py` file that caused children sources that are outside of the bounding box also to be returned. The fix is fairly simple and pushed it in the same branch. Since the fix is simple, I don't know if it can be in the same ticket or has to come from another ticket necessarily because the PR was reverted.