Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-15921

Excessive memory usage in MatchPessimisticB

    XMLWordPrintable

    Details

    • Team:
      Alert Production

      Description

      Hayden Smotherman, always an effective discoverer of bugs, ran into some decam fields that consistently crashed due to excessive memory usage, using >100GB on astro-lsst-01. We reran the failing chip in pdb and broke in when the memory usage suddenly began climbing, which corresponded to PessimisticPatternMatcherB._build_distances_and_angles(). This function creates several N x N arrays, where N is the number of objects in the reference catalog. In the failing case there were 37k reference objects (from Gaia) covering the 2k x 4k sensor, so the arrays total about 50 GB. We did not investigate the additional memory usage beyond this. We did double check that the ref cat sources sent to the matcher were correct and did not include any extra sources beyond the normal chip buffer area. By eye, the field appears dense but not nearly as crowded as something like the DECAPS area. Visit 439117 ccd 7 of the Allen NEO data was our test case.

       

        Attachments

          Issue Links

            Activity

            Hide
            cmorrison Chris Morrison [X] (Inactive) added a comment -

            Work for this ticket is done on DM-16361.

            Show
            cmorrison Chris Morrison [X] (Inactive) added a comment - Work for this ticket is done on DM-16361 .

              People

              Assignee:
              Unassigned Unassigned
              Reporter:
              ctslater Colin Slater
              Watchers:
              Andrew Connolly, Chris Morrison [X] (Inactive), Colin Slater, Eric Bellm, Hayden Smotherman, John Swinbank
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Jenkins Builds

                  No builds found.