Uploaded image for project: 'Request For Comments'
  1. Request For Comments
  2. RFC-635

Add pyarrow as eups-distributed third party package

    Details

    • Type: RFC
    • Status: Implemented
    • Resolution: Done
    • Component/s: DM
    • Labels:
      None

      Description

      Add pyarrow to lsst_distrib as an eup-distributed third party package:
      https://confluence.lsstcorp.org/display/DM/DM+Third+Party+Software

      It has been installed in the shared stacks on lsst-dev
      DM-14107 and the LSP DM-14313. The postprocessing tasks being moved from qa_explorer to pipe_tasks on DM-16234 depends on pyarrow.

        Attachments

          Issue Links

            Activity

            Hide
            ktl Kian-Tat Lim added a comment -

            Putting this in conda means that every "stack" user, regardless of what they are doing, is forced to install it.  pyarrow is low-level enough and useful enough that this is likely worth it.  +1 from me.

            Show
            ktl Kian-Tat Lim added a comment - Putting this in conda means that every "stack" user, regardless of what they are doing, is forced to install it.  pyarrow is low-level enough and useful enough that this is likely worth it.  +1 from me.
            Hide
            yusra Yusra AlSayyad added a comment -

            I'm not sure it's worth updating Confluence

            Agreed, let's remove the link from https://developer.lsst.io/stack/packaging-third-party-eups-dependencies.html which was where I found it

            Putting this in conda means that every "stack" user, regardless of what they are doing, is forced to install it

            As of DM-16234, pipe_tasks will depend on pyarrow. Does this make you nervous; do we need an even higher level package or is pipe_tasks high enough?

            Show
            yusra Yusra AlSayyad added a comment - I'm not sure it's worth updating Confluence Agreed, let's remove the link from https://developer.lsst.io/stack/packaging-third-party-eups-dependencies.html which was where I found it Putting this in conda means that every "stack" user, regardless of what they are doing, is forced to install it As of DM-16234 , pipe_tasks will depend on pyarrow. Does this make you nervous; do we need an even higher level package or is pipe_tasks high enough?
            Hide
            tjenness Tim Jenness added a comment -

            I'm fine with conda although we don't yet have an automated conda environment management scheme in place. Worst case is that we announce on community that if people want to continue using their current env they will have to conda install pyarrow themselves.

            Show
            tjenness Tim Jenness added a comment - I'm fine with conda although we don't yet have an automated conda environment management scheme in place. Worst case is that we announce on community that if people want to continue using their current env they will have to conda install pyarrow themselves.
            Hide
            yusra Yusra AlSayyad added a comment -

            Adopting with implementation ticket to add pyarrow to scipipe_conda_env rather than building third party repo.

            Basing this decision on a couple things:

            Show
            yusra Yusra AlSayyad added a comment - Adopting with implementation ticket to add pyarrow to scipipe_conda_env rather than building third party repo. Basing this decision on a couple things: The use of the word "migrate" in a note in here: https://confluence.lsstcorp.org/pages/viewpage.action?pageId=118587857 It's less work for me than building https://github.com/apache/arrow and extracting pyarrow from the python subdirectory.
            Hide
            swinbank John Swinbank added a comment -

            This has been done (with caveats; see DM-22256).

            Show
            swinbank John Swinbank added a comment - This has been done (with caveats; see DM-22256 ).

              People

              • Assignee:
                yusra Yusra AlSayyad
                Reporter:
                yusra Yusra AlSayyad
                Watchers:
                Brian Van Klaveren, Igor Gaponenko, John Swinbank, Kian-Tat Lim, Tim Jenness, Yusra AlSayyad
              • Votes:
                0 Vote for this issue
                Watchers:
                6 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:
                  Planned End:

                  Summary Panel