Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-22125

Consider upgrading pyarrow to 0.15.1 due to memory leak in 0.13

    XMLWordPrintable

    Details

    • Story Points:
      0.1
    • Team:
      Architecture

      Description

      There's a memory leak in pyarrow 0.13.1 when reading/writing Pandas to parquet.

      https://github.com/apache/arrow/issues/2624

      This is fixed in 0.15.1.

      I'd like to suggest upgrading the required pyarrow in the stack to at least 0.15.1.

      Yusra AlSayyad What do you think?

        Attachments

          Issue Links

            Activity

            Hide
            yusra Yusra AlSayyad added a comment -

            See conversation in #dm-arch. The pyarrow version added to the conda env is incompatible with the eups boost (and is unusable on mac os). We're still working on getting it into the stack. 0.15.1 no prob.

            Show
            yusra Yusra AlSayyad added a comment - See conversation in #dm-arch. The pyarrow version added to the conda env is incompatible with the eups boost (and is unusable on mac os). We're still working on getting it into the stack. 0.15.1 no prob.
            Hide
            tjenness Tim Jenness added a comment -

            I did this as part of DM-22817 general conda env refresh. I have no idea whether it's helped anything so I'll leave this ticket open for now.

            Show
            tjenness Tim Jenness added a comment - I did this as part of DM-22817 general conda env refresh. I have no idea whether it's helped anything so I'll leave this ticket open for now.
            Hide
            swinbank John Swinbank added a comment -

            If this ticket is both literally (as in, we considered it) and practically (as in, we actually did it) done, let's mark it as such — it just seems confusing to hold open an “upgrade to 0.15.1” ticket, when we've already upgraded to 0.15.1.

            If we need another ticket to track down a memory leak, perhaps Michael Wood-Vasey [X] will file it.

            Show
            swinbank John Swinbank added a comment - If this ticket is both literally (as in, we considered it) and practically (as in, we actually did it) done, let's mark it as such — it just seems confusing to hold open an “upgrade to 0.15.1” ticket, when we've already upgraded to 0.15.1. If we need another ticket to track down a memory leak, perhaps Michael Wood-Vasey [X] will file it.
            Hide
            tjenness Tim Jenness added a comment -

            My reticence was mostly over having never even tried the new pyarrow so I don't know if it's at all usable (it's from the conda default channel, not conda-forge channel) – or at least as usable as 0.13 was.

            Show
            tjenness Tim Jenness added a comment - My reticence was mostly over having never even tried the new pyarrow so I don't know if it's at all usable (it's from the conda default channel, not conda-forge channel) – or at least as usable as 0.13 was.

              People

              Assignee:
              tjenness Tim Jenness
              Reporter:
              wmwood-vasey Michael Wood-Vasey [X] (Inactive)
              Watchers:
              Brian Van Klaveren, John Swinbank, Michael Wood-Vasey [X] (Inactive), Tim Jenness, Yusra AlSayyad
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved: