Details
-
Type:
Story
-
Status: Done
-
Resolution: Done
-
Fix Version/s: None
-
Component/s: ap_verify
-
Labels:None
-
Story Points:2
-
Epic Link:
-
Sprint:AP F22-4 (September)
-
Team:Alert Production
-
Urgent?:No
Description
Add the parquet transform tasks used by DRP (such as TransformSourceTableTask and ConsolidateVisitSummaryTask) as a new ap_pipe pipeline, and add the new pipeline to ApVerify.yaml and ApVerifyWithFakes.yaml.
Attachments
Issue Links
- mentioned in
-
Page Loading...
Activity
| Field | Original Value | New Value |
|---|---|---|
| Remote Link | This issue links to "Page (Confluence)" [ 34349 ] |
Krzysztof Findeisen
made changes -
| Status | To Do [ 10001 ] | In Progress [ 3 ] |
Krzysztof Findeisen
made changes -
| Attachment | ApVerify.png [ 63569 ] | |
| Attachment | ApVerifyWithFakes.png [ 63570 ] |
Thanks for agreeing to review this, John Parejko! For reference, I've attached the compiled pipelines to this issue.
Krzysztof Findeisen
added a comment - Thanks for agreeing to review this, John Parejko ! For reference, I've attached the compiled pipelines to this issue. Krzysztof Findeisen
made changes -
| Reviewers | John Parejko [ parejkoj ] | |
| Status | In Progress [ 3 ] | In Review [ 10004 ] |
Krzysztof Findeisen
made changes -
| Component/s | ap_verify [ 14167 ] |
Looks good.
| Status | In Review [ 10004 ] | Reviewed [ 10101 ] |
Krzysztof Findeisen
made changes -
| Resolution | Done [ 10000 ] | |
| Status | Reviewed [ 10101 ] | Done [ 10002 ] |
The first iteration adds DRP tasks that weren't obviously irrelevant to AP: visitSummary, conversions of src and visitSummary to Parquet, and visit-level aggregation of source and *Diff_diaSrcTable. I've also added fakes_ analogues of all of these, whether or not DRP had them.
Unfortunately, nobody seems to know whether these are the datasets we actually want, so I'm not sure how to test this.