# Reimplement make_apdb.py for Gen 3

XMLWordPrintable

#### Details

• Type: Story
• Status: Done
• Resolution: Done
• Fix Version/s: None
• Component/s:
• Labels:
• Story Points:
6
• Sprint:
AP F20-5 (October), AP F20-6 (November)
• Team:
• Urgent?:
No

#### Description

make_apdb.py is a non-Task command-line script. To make it easy to keep database configuration consistent between it and ap_pipe.py, make_apdb.py takes exactly the same config arguments as ap_pipe.py. In Gen 2, it does this by internally reusing ApPipeConfig and two custom argparse.Action classes from CmdLineTask. All of these classes will be removed along with Gen 2.

Develop a new make_apdb.py that matches the interface used to configure the AP Pipeline (or at least the APDB) with the pipetask activator:

 pipetask run -c label:config_var.sub_config=6 pipetask run --config label:config_var.sub_config=6 pipetask run -C label:path pipetask run --config-file label:path pipetask run --pipeline path # May configure any component, including APDB 

It may be useful to wait until DM-21939 is done, so that we can see exactly what the new ap_pipe.py interface will look like. The current make_apdb.py will work with the Gen 3 pipeline for as long as CmdLineTask is also supported.

Unfortunately, the Gen 3 equivalents of ConfigFileAction and ConfigValueAction do not do any config processing, which is instead tightly coupled to the activator code itself. So the Gen 3 version will have to find a different way to give users a consistent UI.

#### Attachments

1. sphinx-docs.tar.gz
209 kB

#### Activity

Hide
Krzysztof Findeisen added a comment -

The new make_apdb.py script should support obs package overrides for the APDB, possibly through apPipe.py or diaPipe.py, as originally proposed on DM-24436.

Show
Krzysztof Findeisen added a comment - The new make_apdb.py script should support obs package overrides for the APDB, possibly through apPipe.py or diaPipe.py , as originally proposed on DM-24436 .
Hide
Krzysztof Findeisen added a comment -

Andy Salnikov mentioned on DM-24584 that pipetask's UI may be changing to favor a purely file-based configuration. Once we're ready to work on this, update the requirements as appropriate.

Show
Krzysztof Findeisen added a comment - Andy Salnikov mentioned on DM-24584 that pipetask 's UI may be changing to favor a purely file-based configuration. Once we're ready to work on this, update the requirements as appropriate.
Hide
Krzysztof Findeisen added a comment - - edited

This ticket ended up changing in spec a lot. After discussion on #dm-alert-prod, I dropped the requirement that the UI for make_apdb.py look like the one for pipetask run. That, in turn, meant there was little point in creating a separate Gen 3 script. Instead, I've altered the existing make_apdb.py to use ApdbConfig directly (a breaking change for anybody calling it) in both Gen 2 and Gen 3.

Meredith Rawls, I suggest looking at the ap_pipe changes commit by commit, since one of the larger commits is a straight copy of existing code from pipe_base.

Show
Krzysztof Findeisen added a comment - - edited This ticket ended up changing in spec a lot. After discussion on #dm-alert-prod , I dropped the requirement that the UI for make_apdb.py look like the one for pipetask run . That, in turn, meant there was little point in creating a separate Gen 3 script. Instead, I've altered the existing make_apdb.py to use ApdbConfig directly (a breaking change for anybody calling it) in both Gen 2 and Gen 3. Meredith Rawls , I suggest looking at the ap_pipe changes commit by commit, since one of the larger commits is a straight copy of existing code from pipe_base .
Hide
Meredith Rawls added a comment -

Thanks for the clear descriptions here, straightforward implementation, and updated docs. Please see my comments on GitHub.

Show
Meredith Rawls added a comment - Thanks for the clear descriptions here, straightforward implementation, and updated docs. Please see my comments on GitHub.
Hide
Krzysztof Findeisen added a comment -

Back In Progress, since Meredith Rawls pointed out that part of this is a problem the Gen 3 people have already solved, and I should use their solution.

Show
Krzysztof Findeisen added a comment - Back In Progress, since Meredith Rawls pointed out that part of this is a problem the Gen 3 people have already solved, and I should use their solution.
Hide
Krzysztof Findeisen added a comment - - edited

Sorry Meredith Rawls, but could you re-review? Other than the corrections you requested, I think the only changes I made were to make_apdb.py (in particular, commit 057d803).

Show
Krzysztof Findeisen added a comment - - edited Sorry Meredith Rawls , but could you re-review? Other than the corrections you requested, I think the only changes I made were to make_apdb.py (in particular, commit 057d803 ).
Hide
Meredith Rawls added a comment -

I think this is good to go now, assuming Jenkins agrees. Thanks for your work on it!

Show
Meredith Rawls added a comment - I think this is good to go now, assuming Jenkins agrees. Thanks for your work on it!

#### People

Assignee:
Krzysztof Findeisen
Reporter:
Krzysztof Findeisen
Reviewers:
Meredith Rawls
Watchers:
Chris Morrison [X] (Inactive), Krzysztof Findeisen, Meredith Rawls