Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-14107

Add pyarrow to shared stack

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      Add pyarrow to the shared stack, to support DM-13793, DM-13770, DM-13876, etc.  Also, if it doesn't break anything else, pandas 0.22 would be a good idea, too (improvements in parquet integration).

        Attachments

          Issue Links

            Activity

            Hide
            tmorton Tim Morton added a comment -

            There must be something funky with the stack-installed pyarrow (not pandas). When I pip install --user --ignore-installed pyarrow, then my test runs fine; when I remove that, it hangs with the signal 11.

            Show
            tmorton Tim Morton added a comment - There must be something funky with the stack-installed pyarrow (not pandas). When I pip install --user --ignore-installed pyarrow , then my test runs fine; when I remove that, it hangs with the signal 11 .
            Hide
            tmorton Tim Morton added a comment -

            Sophie Reed just hit a signal 11 issue trying to use the shared stack and my parquet stuff, too, so there's something that's definitely not right.

            Show
            tmorton Tim Morton added a comment - Sophie Reed just hit a signal 11 issue trying to use the shared stack and my parquet stuff, too, so there's something that's definitely not right.
            Hide
            swinbank John Swinbank added a comment -

            That sounds scarily like an ABI incompatibility creeping in somewhere, but it's very hard to tell where. This is why I'm scared of installing stuff into the stack using Conda.

            I'll play around to see if I can get something which works, but this may be effectively insoluble within the constraints that Conda puts upon us.

            Show
            swinbank John Swinbank added a comment - That sounds scarily like an ABI incompatibility creeping in somewhere, but it's very hard to tell where. This is why I'm scared of installing stuff into the stack using Conda. I'll play around to see if I can get something which works, but this may be effectively insoluble within the constraints that Conda puts upon us.
            Hide
            tmorton Tim Morton added a comment -

            I don't suppose you can pip install pyarrow instead of using conda?

            Show
            tmorton Tim Morton added a comment - I don't suppose you can pip install pyarrow instead of using conda?
            Hide
            swinbank John Swinbank added a comment - - edited

            Per DM-15494, this is now working in /software/lsstsw/stack. I don't propose to spend further effort updating old stacks unless there's a critical need, so I reckon we can close this issue. Tim Morton (or anybody else) please reopen if you disagree.

            Show
            swinbank John Swinbank added a comment - - edited Per DM-15494 , this is now working in /software/lsstsw/stack . I don't propose to spend further effort updating old stacks unless there's a critical need, so I reckon we can close this issue. Tim Morton (or anybody else) please reopen if you disagree.

              People

              • Assignee:
                swinbank John Swinbank
                Reporter:
                tmorton Tim Morton
                Reviewers:
                Tim Morton
                Watchers:
                John Swinbank, Tim Jenness, Tim Morton
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Summary Panel