Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-22125

Consider upgrading pyarrow to 0.15.1 due to memory leak in 0.13

    Details

    • Story Points:
      0.1
    • Team:
      Architecture

      Description

      There's a memory leak in pyarrow 0.13.1 when reading/writing Pandas to parquet.

      https://github.com/apache/arrow/issues/2624

      This is fixed in 0.15.1.

      I'd like to suggest upgrading the required pyarrow in the stack to at least 0.15.1.

      Yusra AlSayyad What do you think?

        Attachments

          Issue Links

            Activity

            Hide
            tjenness Tim Jenness added a comment -

            My reticence was mostly over having never even tried the new pyarrow so I don't know if it's at all usable (it's from the conda default channel, not conda-forge channel) – or at least as usable as 0.13 was.

            Show
            tjenness Tim Jenness added a comment - My reticence was mostly over having never even tried the new pyarrow so I don't know if it's at all usable (it's from the conda default channel, not conda-forge channel) – or at least as usable as 0.13 was.
            Hide
            swinbank John Swinbank added a comment -

            If this ticket is both literally (as in, we considered it) and practically (as in, we actually did it) done, let's mark it as such — it just seems confusing to hold open an “upgrade to 0.15.1” ticket, when we've already upgraded to 0.15.1.

            If we need another ticket to track down a memory leak, perhaps Michael Wood-Vasey will file it.

            Show
            swinbank John Swinbank added a comment - If this ticket is both literally (as in, we considered it) and practically (as in, we actually did it) done, let's mark it as such — it just seems confusing to hold open an “upgrade to 0.15.1” ticket, when we've already upgraded to 0.15.1. If we need another ticket to track down a memory leak, perhaps Michael Wood-Vasey will file it.
            Hide
            tjenness Tim Jenness added a comment -

            I did this as part of DM-22817 general conda env refresh. I have no idea whether it's helped anything so I'll leave this ticket open for now.

            Show
            tjenness Tim Jenness added a comment - I did this as part of DM-22817 general conda env refresh. I have no idea whether it's helped anything so I'll leave this ticket open for now.
            Hide
            yusra Yusra AlSayyad added a comment -

            See conversation in #dm-arch. The pyarrow version added to the conda env is incompatible with the eups boost (and is unusable on mac os). We're still working on getting it into the stack. 0.15.1 no prob.

            Show
            yusra Yusra AlSayyad added a comment - See conversation in #dm-arch. The pyarrow version added to the conda env is incompatible with the eups boost (and is unusable on mac os). We're still working on getting it into the stack. 0.15.1 no prob.

              People

              • Assignee:
                tjenness Tim Jenness
                Reporter:
                wmwood-vasey Michael Wood-Vasey
                Watchers:
                Brian Van Klaveren, John Swinbank, Michael Wood-Vasey, Tim Jenness, Yusra AlSayyad
              • Votes:
                0 Vote for this issue
                Watchers:
                5 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Summary Panel