Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-15578

lsst.afw.table.BaseCatalog can use 50x the memory of the underlying data

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Won't Fix
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: afw
    • Labels:
      None
    • Team:
      Data Release Production

      Description

      I was running out of memory when using fits files persisted with lsst.afw.table.BaseCatalog with millions of records and relatively narrow rows (a few doubles and ints). The total file sizes were 200Mb - 5Gb, but after reading in with lsst.afw.table.BaseCatalog.readFits() the tables were using 10Gb - 40Gb. This problem makes it impossible to run fgcmcal on any dataset larger than HSC PDR1.
      A possible workaround is to copy out the table to numpy arrays, but it is not clear if setting the lsst.afw.table.BaseCatalog to None actually clears the memory. I have attached a test file to the ticket which demonstrates the problem. In this case, it creates a 24Mb fits table that nevertheless takes up ~200Mb of RAM as an lsst.afw.table.BaseCatalog.

        Attachments

          Issue Links

          blocks

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-15545 Run FGCM on HSC S18A

          • Done
          mentioned in

          Page Loading...

            Activity

            Ascending order - Click to sort in descending order
            Hide
            Permalink
            jbosch Jim Bosch added a comment -

            I'm calling this Won't Fix. It's a real problem, but if my guess at the cause is right, it's pretty inherent in the afw.table design, and the right solution is to find ways to not use afw.table for very tall, very narrow tables (as I suspect Eli Rykoff has already done).

            Show
            jbosch Jim Bosch added a comment - I'm calling this Won't Fix. It's a real problem, but if my guess at the cause is right, it's pretty inherent in the afw.table design, and the right solution is to find ways to not use afw.table for very tall, very narrow tables (as I suspect Eli Rykoff has already done).
            Hide
            Permalink
            erykoff Eli Rykoff added a comment -

            Ah! This must be related to my table-append-bakeoff! I discovered the same problem more than once! And have figured out how to work around it, yes.

            Show
            erykoff Eli Rykoff added a comment - Ah! This must be related to my table-append-bakeoff! I discovered the same problem more than once! And have figured out how to work around it, yes.

              People

              Assignee:
              jbosch Jim Bosch
              Reporter:
              erykoff Eli Rykoff
              Watchers:
              Eli Rykoff, Jim Bosch
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Jenkins

                  No builds found.