Details
-
Type:
Bug
-
Status: Invalid
-
Resolution: Done
-
Fix Version/s: None
-
Component/s: meas_astrom
-
Labels:
-
Team:Alert Production
Description
Hayden Smotherman, always an effective discoverer of bugs, ran into some decam fields that consistently crashed due to excessive memory usage, using >100GB on astro-lsst-01. We reran the failing chip in pdb and broke in when the memory usage suddenly began climbing, which corresponded to PessimisticPatternMatcherB._build_distances_and_angles(). This function creates several N x N arrays, where N is the number of objects in the reference catalog. In the failing case there were 37k reference objects (from Gaia) covering the 2k x 4k sensor, so the arrays total about 50 GB. We did not investigate the additional memory usage beyond this. We did double check that the ref cat sources sent to the matcher were correct and did not include any extra sources beyond the normal chip buffer area. By eye, the field appears dense but not nearly as crowded as something like the DECAPS area. Visit 439117 ccd 7 of the Allen NEO data was our test case.
Attachments
Issue Links
- is triggering
-
DM-16361 Optimize memory usage in MatchPessimisticB
- Done
Work for this ticket is done on
DM-16361.