Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-22125

Consider upgrading pyarrow to 0.15.1 due to memory leak in 0.13

    Details

    • Story Points:
      0.1
    • Team:
      Architecture

      Description

      There's a memory leak in pyarrow 0.13.1 when reading/writing Pandas to parquet.

      https://github.com/apache/arrow/issues/2624

      This is fixed in 0.15.1.

      I'd like to suggest upgrading the required pyarrow in the stack to at least 0.15.1.

      Yusra AlSayyad What do you think?

        Attachments

          Issue Links

            Activity

            wmwood-vasey Michael Wood-Vasey created issue -
            wmwood-vasey Michael Wood-Vasey made changes -
            Field Original Value New Value
            Link This issue relates to DM-21746 [ DM-21746 ]
            yusra Yusra AlSayyad made changes -
            Watchers Michael Wood-Vasey, Yusra AlSayyad [ Michael Wood-Vasey, Yusra AlSayyad ] Brian Van Klaveren, John Swinbank, Michael Wood-Vasey, Tim Jenness, Yusra AlSayyad [ Brian Van Klaveren, John Swinbank, Michael Wood-Vasey, Tim Jenness, Yusra AlSayyad ]
            swinbank John Swinbank made changes -
            Component/s Science Pipelines [ 10706 ]
            swinbank John Swinbank made changes -
            Team Data Release Production [ 10301 ]
            yusra Yusra AlSayyad made changes -
            Link This issue relates to DM-22256 [ DM-22256 ]
            tjenness Tim Jenness made changes -
            Link This issue is triggering RFC-664 [ RFC-664 ]
            tjenness Tim Jenness made changes -
            Link This issue is triggering DM-22817 [ DM-22817 ]
            swinbank John Swinbank made changes -
            Status To Do [ 10001 ] In Progress [ 3 ]
            swinbank John Swinbank made changes -
            Resolution Done [ 10000 ]
            Status In Progress [ 3 ] Done [ 10002 ]
            tjenness Tim Jenness made changes -
            Assignee Yusra AlSayyad [ yusra ] Tim Jenness [ tjenness ]
            tjenness Tim Jenness made changes -
            Team Data Release Production [ 10301 ] Architecture [ 10304 ]
            tjenness Tim Jenness made changes -
            Story Points 1 0.1
            tjenness Tim Jenness made changes -
            Component/s Developer Infrastructure [ 10712 ]

              People

              • Assignee:
                tjenness Tim Jenness
                Reporter:
                wmwood-vasey Michael Wood-Vasey
                Watchers:
                Brian Van Klaveren, John Swinbank, Michael Wood-Vasey, Tim Jenness, Yusra AlSayyad
              • Votes:
                0 Vote for this issue
                Watchers:
                5 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Summary Panel