Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-27008

Update Functor implementation to use gen3 butler/parquet access

    XMLWordPrintable

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None
    • Story Points:
      8
    • Epic Link:
    • Team:
      Data Release Production
    • Urgent?:
      No

      Description

      Gen3 parquet access is fundamentally different from gen2.  In gen2, a butler.get  call on a parquet dataset call returned a lazy ParquetTable object that doesn't load any data until requested by .getDataFrame at which point specific columns can be requested.  However, in gen3, a butler.get call returns data directly, and specific columns are requested at this time.  

      The Functor implementation currently in pipe_tasks is based on the gen2 ParquetTable object, as it takes one of those as the input to its _call_ method.  Updating for gen3 butler will need to change this.  

      A related concern is that one of the reasons for the gen2 ParquetTable implementation to be the way it was was to not have to read parquet metadata with every data load.  Part of this ticket should be to explore the degree to which this metadata loading affects practical use cases, and if anything needs to be changed in the gen3 parquet-interaction layer.

        Attachments

          Issue Links

            Activity

              People

              Assignee:
              tmorton Tim Morton [X] (Inactive)
              Reporter:
              tmorton Tim Morton [X] (Inactive)
              Reviewers:
              Lauren MacArthur
              Watchers:
              Jim Bosch, Kian-Tat Lim, Lauren MacArthur, Sophie Reed, Tim Jenness, Tim Morton [X] (Inactive), Yusra AlSayyad
              Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Jenkins

                  No builds found.