Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-27008

Update Functor implementation to use gen3 butler/parquet access

    XMLWordPrintable

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None
    • Story Points:
      8
    • Epic Link:
    • Team:
      Data Release Production
    • Urgent?:
      No

      Description

      Gen3 parquet access is fundamentally different from gen2.  In gen2, a butler.get  call on a parquet dataset call returned a lazy ParquetTable object that doesn't load any data until requested by .getDataFrame at which point specific columns can be requested.  However, in gen3, a butler.get call returns data directly, and specific columns are requested at this time.  

      The Functor implementation currently in pipe_tasks is based on the gen2 ParquetTable object, as it takes one of those as the input to its _call_ method.  Updating for gen3 butler will need to change this.  

      A related concern is that one of the reasons for the gen2 ParquetTable implementation to be the way it was was to not have to read parquet metadata with every data load.  Part of this ticket should be to explore the degree to which this metadata loading affects practical use cases, and if anything needs to be changed in the gen3 parquet-interaction layer.

        Attachments

          Issue Links

            Activity

            tmorton Tim Morton [X] (Inactive) created issue -
            yusra Yusra AlSayyad made changes -
            Field Original Value New Value
            Epic Link DM-26790 [ 439740 ]
            tmorton Tim Morton [X] (Inactive) made changes -
            Status To Do [ 10001 ] In Progress [ 3 ]
            tmorton Tim Morton [X] (Inactive) made changes -
            Reviewers Sophie Reed [ sophiereed ]
            Status In Progress [ 3 ] In Review [ 10004 ]
            sophiereed Sophie Reed made changes -
            Status In Review [ 10004 ] Reviewed [ 10101 ]
            yusra Yusra AlSayyad made changes -
            Epic Link DM-26790 [ 439740 ] DM-27975 [ 442760 ]
            tmorton Tim Morton [X] (Inactive) made changes -
            Resolution Done [ 10000 ]
            Status Reviewed [ 10101 ] Done [ 10002 ]
            tmorton Tim Morton [X] (Inactive) made changes -
            Story Points 8
            ktl Kian-Tat Lim made changes -
            Watchers Jim Bosch, Kian-Tat Lim, Sophie Reed, Tim Jenness, Tim Morton, Yusra AlSayyad [ Jim Bosch, Kian-Tat Lim, Sophie Reed, Tim Jenness, Tim Morton, Yusra AlSayyad ] Jim Bosch, Kian-Tat Lim, Lauren MacArthur, Sophie Reed, Tim Jenness, Tim Morton, Yusra AlSayyad [ Jim Bosch, Kian-Tat Lim, Lauren MacArthur, Sophie Reed, Tim Jenness, Tim Morton, Yusra AlSayyad ]
            ktl Kian-Tat Lim made changes -
            Resolution Done [ 10000 ]
            Status Done [ 10002 ] To Do [ 10001 ]
            lskelvin Lee Kelvin made changes -
            Link This issue relates to DM-27255 [ DM-27255 ]
            tmorton Tim Morton [X] (Inactive) made changes -
            Status To Do [ 10001 ] In Progress [ 3 ]
            tmorton Tim Morton [X] (Inactive) made changes -
            Reviewers Sophie Reed [ sophiereed ] Lauren MacArthur [ lauren ]
            Status In Progress [ 3 ] In Review [ 10004 ]
            lauren Lauren MacArthur made changes -
            Status In Review [ 10004 ] Reviewed [ 10101 ]
            tmorton Tim Morton [X] (Inactive) made changes -
            Resolution Done [ 10000 ]
            Status Reviewed [ 10101 ] Done [ 10002 ]
            tmorton Tim Morton [X] (Inactive) made changes -
            Link This issue is triggering DM-28922 [ DM-28922 ]

              People

              Assignee:
              tmorton Tim Morton [X] (Inactive)
              Reporter:
              tmorton Tim Morton [X] (Inactive)
              Reviewers:
              Lauren MacArthur
              Watchers:
              Jim Bosch, Kian-Tat Lim, Lauren MacArthur, Sophie Reed, Tim Jenness, Tim Morton [X] (Inactive), Yusra AlSayyad
              Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Jenkins

                  No builds found.