Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-36198

Add parquet transform tasks to ap_verify

    XMLWordPrintable

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: ap_verify
    • Labels:
      None

      Description

      Add the parquet transform tasks used by DRP (such as TransformSourceTableTask and ConsolidateVisitSummaryTask) as a new ap_pipe pipeline, and add the new pipeline to ApVerify.yaml and ApVerifyWithFakes.yaml.

        Attachments

          Issue Links

          mentioned in

          Page Loading...

            Activity

            Ascending order - Click to sort in descending order
            Hide
            Permalink
            krzys Krzysztof Findeisen added a comment -

            The first iteration adds DRP tasks that weren't obviously irrelevant to AP: visitSummary, conversions of src and visitSummary to Parquet, and visit-level aggregation of source and *Diff_diaSrcTable. I've also added fakes_ analogues of all of these, whether or not DRP had them.

            Unfortunately, nobody seems to know whether these are the datasets we actually want, so I'm not sure how to test this.

            Show
            krzys Krzysztof Findeisen added a comment - The first iteration adds DRP tasks that weren't obviously irrelevant to AP: visitSummary , conversions of src and visitSummary to Parquet, and visit-level aggregation of source and *Diff_diaSrcTable . I've also added fakes_ analogues of all of these, whether or not DRP had them. Unfortunately, nobody seems to know whether these are the datasets we actually want, so I'm not sure how to test this.
            Hide
            Permalink
            krzys Krzysztof Findeisen added a comment -

            Thanks for agreeing to review this, John Parejko! For reference, I've attached the compiled pipelines to this issue.

            Show
            krzys Krzysztof Findeisen added a comment - Thanks for agreeing to review this, John Parejko ! For reference, I've attached the compiled pipelines to this issue.
            Hide
            Permalink
            Parejkoj John Parejko added a comment -

            Looks good.

            Show
            Parejkoj John Parejko added a comment - Looks good.

              People

              Assignee:
              krzys Krzysztof Findeisen
              Reporter:
              sullivan Ian Sullivan
              Reviewers:
              John Parejko
              Watchers:
              Ian Sullivan, John Parejko, Krzysztof Findeisen
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Jenkins

                  No builds found.