Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-34875

Migrate PipelineTasks with DataFrame connections to ArrowTable

    XMLWordPrintable

    Details

    • Type: Story
    • Status: To Do
    • Resolution: Unresolved
    • Fix Version/s: None
    • Component/s: pipe_tasks
    • Labels:
      None
    • Team:
      Data Release Production
    • Urgent?:
      No

      Description

      Implements the second step of RFC-844, in which tasks that use DataFrame connections are changed to use ArrowTable instead, converting to/from pandas internally where needed.

      This may be an umbrella ticket for work done on other tickets; it should not be closed until no DataFrame tasks exist, unless a new implementation ticket is added to RFC-844 to cover any that remain.

        Attachments

          Issue Links

          is triggered by

          RFC - An issue whose primary purpose is to advertise an intent and allow stakeholders to comment. RFC-844 Use pyarrow.Table for butler parquet datasets instead of pandas.DataFrame

          • Adopted

            Activity

            Ascending order - Click to sort in descending order
            jbosch Jim Bosch created issue -
            jbosch Jim Bosch made changes -
            Field Original Value New Value
            Link This issue is triggered by RFC-844 [ RFC-844 ]

              People

              Assignee:
              jbosch Jim Bosch
              Reporter:
              jbosch Jim Bosch
              Watchers:
              Eric Bellm, Jim Bosch
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

                Dates

                Created:
                Updated:

                  Jenkins

                  No builds found.