Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-22062

Add parquet support to Gen3 Butler

    XMLWordPrintable

    Details

      Description

      Port formatters for Parquet files to Gen3, including support for lazy loading of columns.

        Attachments

          Issue Links

            Activity

            Hide
            jbosch Jim Bosch added a comment -

            I'm pretty confident the behavior here is correct, given the tests I've written.  I'm less confident that it's (all of) the behavior we need, and even less confident that the documentation uses the right terminology for Pandas things.

             

            Show
            jbosch Jim Bosch added a comment - I'm pretty confident the behavior here is correct, given the tests I've written.  I'm less confident that it's (all of) the behavior we need, and even less confident that the documentation uses the right terminology for Pandas things.  
            Hide
            yusra Yusra AlSayyad added a comment -

            I'm not confident that it's all the behavior we need either, but that will be revealed when I move on to writing the Gen3 versions of the writeObjectTable and transformCatalog tasks this afternoon. I'm OK with your merging this now if you open up a new ticket to address stuff that comes up when using it.

            Show
            yusra Yusra AlSayyad added a comment - I'm not confident that it's all the behavior we need either, but that will be revealed when I move on to writing the Gen3 versions of the writeObjectTable and transformCatalog tasks this afternoon. I'm OK with your merging this now if you open up a new ticket to address stuff that comes up when using it.

              People

              Assignee:
              jbosch Jim Bosch
              Reporter:
              jbosch Jim Bosch
              Reviewers:
              Yusra AlSayyad
              Watchers:
              Jim Bosch, Yusra AlSayyad
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  CI Builds

                  No builds found.