Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-29936

Enable getting Children without repeatedly checking if the SourceCatalog is sorted

    XMLWordPrintable

Details

    • Improvement
    • Status: Won't Fix
    • Resolution: Done
    • None
    • None
    • None

    Description

      While fred3m was chasing the errors in DM-29907, it was realized that `afwTable.SourceCatalog` forbids getting children if the catalog is unsorted. In checking if the catalog is sorted, it performs an operation that is O( n), which is as expensive as fetching the children even from an unsorted catalog. Moreover, this check is performed for each parent, thereby making getting the children for all parents an O(n^2) operation. However, it is desirable to check (somewhere, once) if the catalog is sorted, as "unsortedness" could indicate a potentially serious problem elsewhere. This ticket captures potential solutions, and discussions made towards that on Slack.

      Attachments

        Issue Links

          Activity

            I think I've fixed it now. There was a logic bug in the `subset` method `references.py` file that caused children sources that are outside of the bounding box also to be returned. The fix is fairly simple and pushed it in the same branch. Since the fix is simple, I don't know if it can be in the same ticket or has to come from another ticket necessarily because the PR was reverted.

            kannawad Arun Kannawadi added a comment - I think I've fixed it now. There was a logic bug in the `subset` method `references.py` file that caused children sources that are outside of the bounding box also to be returned. The fix is fairly simple and pushed it in the same branch. Since the fix is simple, I don't know if it can be in the same ticket or has to come from another ticket necessarily because the PR was reverted.

            Weirdly, ci_hsc_gen3 doesn't have build issues, only ci_hsc_gen2 does. Is this a concern?

            kannawad Arun Kannawadi added a comment - Weirdly, ci_hsc_gen3 doesn't have build issues, only ci_hsc_gen2 does. Is this a concern?
            jbosch Jim Bosch added a comment -

            This happens rarely enough that we don't have a policy (AFAIK) for whether to make it another ticket, but I would recommend another ticket just to avoid various kinds of clashes complicating things.

            There are some things ci_hsc_gen2 does that ci_hsc_gen3 does not, and they take different code paths in much of the I/O.  I wouldn't have expected them to differ on this ticket, but that's not a strong expectation and I wouldn't be able to say more without looking much more closely.

            jbosch Jim Bosch added a comment - This happens rarely enough that we don't have a policy (AFAIK) for whether to make it another ticket, but I would recommend another ticket just to avoid various kinds of clashes complicating things. There are some things ci_hsc_gen2 does that ci_hsc_gen3 does not, and they take different code paths in much of the I/O.  I wouldn't have expected them to differ on this ticket, but that's not a strong expectation and I wouldn't be able to say more without looking much more closely.

            Alright. Marking this as Won't Fix now and will create a new PR with the patch included in DM-30105.

            kannawad Arun Kannawadi added a comment - Alright. Marking this as Won't Fix now and will create a new PR with the patch included in DM-30105 .

            To conclude the gen2/gen3 differences in this ticket: in forcedPhotCcd.py, ci_hsc_gen2 gets the `refCat` by calling the `fetchReferences` method from `runDataRef` whereas, ci_hsc_gen3 gets the same by calling `mergeAndFilterReferences` method in `runQuantum`. Only the first method takes the code path that contained the bug, so it is perfectly reasonable that ci_hsc_gen3 build passed but not ci_hsc_gen2.

            kannawad Arun Kannawadi added a comment - To conclude the gen2/gen3 differences in this ticket: in forcedPhotCcd.py, ci_hsc_gen2 gets the `refCat` by calling the `fetchReferences` method from `runDataRef` whereas, ci_hsc_gen3 gets the same by calling `mergeAndFilterReferences` method in `runQuantum`. Only the first method takes the code path that contained the bug, so it is perfectly reasonable that ci_hsc_gen3 build passed but not ci_hsc_gen2.

            People

              kannawad Arun Kannawadi
              kannawad Arun Kannawadi
              Fred Moolekamp
              Arun Kannawadi, Fred Moolekamp, Jim Bosch
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Jenkins

                  No builds found.