Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-15921

Excessive memory usage in MatchPessimisticB

    XMLWordPrintable

    Details

    • Team:
      Alert Production

      Description

      Hayden Smotherman, always an effective discoverer of bugs, ran into some decam fields that consistently crashed due to excessive memory usage, using >100GB on astro-lsst-01. We reran the failing chip in pdb and broke in when the memory usage suddenly began climbing, which corresponded to PessimisticPatternMatcherB._build_distances_and_angles(). This function creates several N x N arrays, where N is the number of objects in the reference catalog. In the failing case there were 37k reference objects (from Gaia) covering the 2k x 4k sensor, so the arrays total about 50 GB. We did not investigate the additional memory usage beyond this. We did double check that the ref cat sources sent to the matcher were correct and did not include any extra sources beyond the normal chip buffer area. By eye, the field appears dense but not nearly as crowded as something like the DECAPS area. Visit 439117 ccd 7 of the Allen NEO data was our test case.

       

        Attachments

          Issue Links

            Activity

            No builds found.
            ctslater Colin Slater created issue -
            ctslater Colin Slater made changes -
            Field Original Value New Value
            Risk Score 0
            smotherh Hayden Smotherman made changes -
            Watchers Andrew Connolly, Bryce Kalmbach, Chris Morrison, Colin Slater, Eric Bellm, John Swinbank [ Andrew Connolly, Bryce Kalmbach, Chris Morrison, Colin Slater, Eric Bellm, John Swinbank ] Andrew Connolly, Bryce Kalmbach, Chris Morrison, Colin Slater, Eric Bellm, Hayden Smotherman, John Swinbank [ Andrew Connolly, Bryce Kalmbach, Chris Morrison, Colin Slater, Eric Bellm, Hayden Smotherman, John Swinbank ]
            swinbank John Swinbank made changes -
            Link This issue is triggering DM-16361 [ DM-16361 ]
            Hide
            cmorrison Chris Morrison [X] (Inactive) added a comment -

            Work for this ticket is done on DM-16361.

            Show
            cmorrison Chris Morrison [X] (Inactive) added a comment - Work for this ticket is done on DM-16361 .
            cmorrison Chris Morrison [X] (Inactive) made changes -
            Resolution Done [ 10000 ]
            Status To Do [ 10001 ] Invalid [ 11005 ]

              People

              Assignee:
              Unassigned Unassigned
              Reporter:
              ctslater Colin Slater
              Watchers:
              Andrew Connolly, Chris Morrison [X] (Inactive), Colin Slater, Eric Bellm, Hayden Smotherman, John Swinbank
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Jenkins

                  No builds found.