Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-22125

Consider upgrading pyarrow to 0.15.1 due to memory leak in 0.13

    Details

    • Story Points:
      0.1
    • Team:
      Architecture

      Description

      There's a memory leak in pyarrow 0.13.1 when reading/writing Pandas to parquet.

      https://github.com/apache/arrow/issues/2624

      This is fixed in 0.15.1.

      I'd like to suggest upgrading the required pyarrow in the stack to at least 0.15.1.

      Yusra AlSayyad What do you think?

        Attachments

          Issue Links

            Activity

            wmwood-vasey Michael Wood-Vasey created issue -
            wmwood-vasey Michael Wood-Vasey made changes -
            Field Original Value New Value
            Link This issue relates to DM-21746 [ DM-21746 ]
            Hide
            yusra Yusra AlSayyad added a comment -

            See conversation in #dm-arch. The pyarrow version added to the conda env is incompatible with the eups boost (and is unusable on mac os). We're still working on getting it into the stack. 0.15.1 no prob.

            Show
            yusra Yusra AlSayyad added a comment - See conversation in #dm-arch. The pyarrow version added to the conda env is incompatible with the eups boost (and is unusable on mac os). We're still working on getting it into the stack. 0.15.1 no prob.
            yusra Yusra AlSayyad made changes -
            Watchers Michael Wood-Vasey, Yusra AlSayyad [ Michael Wood-Vasey, Yusra AlSayyad ] Brian Van Klaveren, John Swinbank, Michael Wood-Vasey, Tim Jenness, Yusra AlSayyad [ Brian Van Klaveren, John Swinbank, Michael Wood-Vasey, Tim Jenness, Yusra AlSayyad ]
            swinbank John Swinbank made changes -
            Component/s Science Pipelines [ 10706 ]
            swinbank John Swinbank made changes -
            Team Data Release Production [ 10301 ]
            yusra Yusra AlSayyad made changes -
            Link This issue relates to DM-22256 [ DM-22256 ]
            tjenness Tim Jenness made changes -
            Link This issue is triggering RFC-664 [ RFC-664 ]
            tjenness Tim Jenness made changes -
            Link This issue is triggering DM-22817 [ DM-22817 ]
            Hide
            tjenness Tim Jenness added a comment -

            I did this as part of DM-22817 general conda env refresh. I have no idea whether it's helped anything so I'll leave this ticket open for now.

            Show
            tjenness Tim Jenness added a comment - I did this as part of DM-22817 general conda env refresh. I have no idea whether it's helped anything so I'll leave this ticket open for now.
            Hide
            swinbank John Swinbank added a comment -

            If this ticket is both literally (as in, we considered it) and practically (as in, we actually did it) done, let's mark it as such — it just seems confusing to hold open an “upgrade to 0.15.1” ticket, when we've already upgraded to 0.15.1.

            If we need another ticket to track down a memory leak, perhaps Michael Wood-Vasey will file it.

            Show
            swinbank John Swinbank added a comment - If this ticket is both literally (as in, we considered it) and practically (as in, we actually did it) done, let's mark it as such — it just seems confusing to hold open an “upgrade to 0.15.1” ticket, when we've already upgraded to 0.15.1. If we need another ticket to track down a memory leak, perhaps Michael Wood-Vasey will file it.
            swinbank John Swinbank made changes -
            Status To Do [ 10001 ] In Progress [ 3 ]
            swinbank John Swinbank made changes -
            Resolution Done [ 10000 ]
            Status In Progress [ 3 ] Done [ 10002 ]
            Hide
            tjenness Tim Jenness added a comment -

            My reticence was mostly over having never even tried the new pyarrow so I don't know if it's at all usable (it's from the conda default channel, not conda-forge channel) – or at least as usable as 0.13 was.

            Show
            tjenness Tim Jenness added a comment - My reticence was mostly over having never even tried the new pyarrow so I don't know if it's at all usable (it's from the conda default channel, not conda-forge channel) – or at least as usable as 0.13 was.
            tjenness Tim Jenness made changes -
            Assignee Yusra AlSayyad [ yusra ] Tim Jenness [ tjenness ]
            tjenness Tim Jenness made changes -
            Team Data Release Production [ 10301 ] Architecture [ 10304 ]
            tjenness Tim Jenness made changes -
            Story Points 1 0.1
            tjenness Tim Jenness made changes -
            Component/s Developer Infrastructure [ 10712 ]

              People

              • Assignee:
                tjenness Tim Jenness
                Reporter:
                wmwood-vasey Michael Wood-Vasey
                Watchers:
                Brian Van Klaveren, John Swinbank, Michael Wood-Vasey, Tim Jenness, Yusra AlSayyad
              • Votes:
                0 Vote for this issue
                Watchers:
                5 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Summary Panel