# Can't rerun ap_verify on same repository in Gen 3 (II)

XMLWordPrintable

#### Details

• Type: Bug
• Status: Done
• Resolution: Done
• Fix Version/s: None
• Component/s:
• Labels:
None
• Story Points:
5
• Sprint:
AP S21-2 (January), AP S21-3 (February)
• Team:
• Urgent?:
No

#### Description

Attempting to rerun ap_verify twice in the same location gives the following error:

 ValueError: Output run 'ap_verify-output' already exists, but --extend-run was not given. 

Reconfigure ap_verify to use current pipetask idioms.

#### Activity

Hide
Krzysztof Findeisen added a comment - - edited

See DMTN-167, as updated by RFC-741, for a discussion of how --output, --output-run, and --extend-run were meant to be used.

Per DM-18715, should also allow for config changes. Currently this means using --output, possibly with --replace-run. This, in turn, would mean either that ap_verify needs to know whether it's already been run in the same workspace, or that it needs to autonomously set up the same type of chained collection as expected by pipetask run.

Show
Krzysztof Findeisen added a comment - - edited See DMTN-167, as updated by RFC-741 , for a discussion of how --output , --output-run , and --extend-run were meant to be used. Per DM-18715 , should also allow for config changes. Currently this means using --output , possibly with --replace-run . This, in turn, would mean either that ap_verify needs to know whether it's already been run in the same workspace, or that it needs to autonomously set up the same type of chained collection as expected by pipetask run .
Hide
Krzysztof Findeisen added a comment -

Per discussion on #dm-alert-prod, rerun support is mainly desirable for restarting interrupted pipelines. I'm therefore not going to try to make APDB operations (the last step in the pipeline) fully rerunnable, as pipeline failures generally happen before this step, and ensuring self-consistent handling of the database is hard to do at a high level in both Gen 2 and Gen 3. It looks like this case was not supported in the original Gen 2 ap_verify, either.

Show
Krzysztof Findeisen added a comment - Per discussion on #dm-alert-prod , rerun support is mainly desirable for restarting interrupted pipelines. I'm therefore not going to try to make APDB operations (the last step in the pipeline) fully rerunnable, as pipeline failures generally happen before this step, and ensuring self-consistent handling of the database is hard to do at a high level in both Gen 2 and Gen 3. It looks like this case was not supported in the original Gen 2 ap_verify , either.
Hide
Krzysztof Findeisen added a comment -

Full rerun capabilities under all circumstances are blocked by a bug in pipetask run --clobber-partial-outputs; see DM-27492 for details.

Show
Krzysztof Findeisen added a comment - Full rerun capabilities under all circumstances are blocked by a bug in pipetask run --clobber-partial-outputs ; see DM-27492 for details.
Hide
Krzysztof Findeisen added a comment -

I've got support for repeating ap_verify runs that fail before they touch the APDB, which I think was the original level of rerunnability. I've also implemented an analogue of DM-18715 for Gen 3 (the --clean-run flag); by default, Gen 3 processing will reuse old runs, which requires the config(s) to match.

Show
Krzysztof Findeisen added a comment - I've got support for repeating ap_verify runs that fail before they touch the APDB, which I think was the original level of rerunnability. I've also implemented an analogue of DM-18715 for Gen 3 (the --clean-run flag); by default, Gen 3 processing will reuse old runs, which requires the config(s) to match.
Hide
Chris Morrison [X] (Inactive) added a comment -

Looks good. Assuming it's clearing Jenkins and running on the current HSC cosmos dataset, go for the merge.

Show
Chris Morrison [X] (Inactive) added a comment - Looks good. Assuming it's clearing Jenkins and running on the current HSC cosmos dataset, go for the merge.

#### People

Assignee:
Krzysztof Findeisen
Reporter:
Krzysztof Findeisen
Reviewers:
Chris Morrison [X] (Inactive)
Watchers:
Chris Morrison [X] (Inactive), Krzysztof Findeisen