Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-22663

Reimplement make_apdb.py for Gen 3

    XMLWordPrintable

    Details

    • Story Points:
      6
    • Sprint:
      AP F20-5 (October), AP F20-6 (November)
    • Team:
      Alert Production
    • Urgent?:
      No

      Description

      make_apdb.py is a non-Task command-line script. To make it easy to keep database configuration consistent between it and ap_pipe.py, make_apdb.py takes exactly the same config arguments as ap_pipe.py. In Gen 2, it does this by internally reusing ApPipeConfig and two custom argparse.Action classes from CmdLineTask. All of these classes will be removed along with Gen 2.

      Develop a new make_apdb.py that matches the interface used to configure the AP Pipeline (or at least the APDB) with the pipetask activator:

      pipetask run -c label:config_var.sub_config=6
      pipetask run --config label:config_var.sub_config=6
      pipetask run -C label:path
      pipetask run --config-file label:path
      pipetask run --pipeline path  # May configure any component, including APDB
      

      It may be useful to wait until DM-21939 is done, so that we can see exactly what the new ap_pipe.py interface will look like. The current make_apdb.py will work with the Gen 3 pipeline for as long as CmdLineTask is also supported.

      Unfortunately, the Gen 3 equivalents of ConfigFileAction and ConfigValueAction do not do any config processing, which is instead tightly coupled to the activator code itself. So the Gen 3 version will have to find a different way to give users a consistent UI.

        Attachments

          Issue Links

            Activity

            Hide
            krzys Krzysztof Findeisen added a comment -

            The new make_apdb.py script should support obs package overrides for the APDB, possibly through apPipe.py or diaPipe.py, as originally proposed on DM-24436.

            Show
            krzys Krzysztof Findeisen added a comment - The new make_apdb.py script should support obs package overrides for the APDB, possibly through apPipe.py or diaPipe.py , as originally proposed on DM-24436 .
            Hide
            krzys Krzysztof Findeisen added a comment -

            Andy Salnikov mentioned on DM-24584 that pipetask's UI may be changing to favor a purely file-based configuration. Once we're ready to work on this, update the requirements as appropriate.

            Show
            krzys Krzysztof Findeisen added a comment - Andy Salnikov mentioned on DM-24584 that pipetask 's UI may be changing to favor a purely file-based configuration. Once we're ready to work on this, update the requirements as appropriate.
            Hide
            krzys Krzysztof Findeisen added a comment - - edited

            This ticket ended up changing in spec a lot. After discussion on #dm-alert-prod, I dropped the requirement that the UI for make_apdb.py look like the one for pipetask run. That, in turn, meant there was little point in creating a separate Gen 3 script. Instead, I've altered the existing make_apdb.py to use ApdbConfig directly (a breaking change for anybody calling it) in both Gen 2 and Gen 3.

            Meredith Rawls, I suggest looking at the ap_pipe changes commit by commit, since one of the larger commits is a straight copy of existing code from pipe_base.

            Show
            krzys Krzysztof Findeisen added a comment - - edited This ticket ended up changing in spec a lot. After discussion on #dm-alert-prod , I dropped the requirement that the UI for make_apdb.py look like the one for pipetask run . That, in turn, meant there was little point in creating a separate Gen 3 script. Instead, I've altered the existing make_apdb.py to use ApdbConfig directly (a breaking change for anybody calling it) in both Gen 2 and Gen 3. Meredith Rawls , I suggest looking at the ap_pipe changes commit by commit, since one of the larger commits is a straight copy of existing code from pipe_base .
            Hide
            mrawls Meredith Rawls added a comment -

            Thanks for the clear descriptions here, straightforward implementation, and updated docs. Please see my comments on GitHub. 

            Show
            mrawls Meredith Rawls added a comment - Thanks for the clear descriptions here, straightforward implementation, and updated docs. Please see my comments on GitHub. 
            Hide
            krzys Krzysztof Findeisen added a comment -

            Back In Progress, since Meredith Rawls pointed out that part of this is a problem the Gen 3 people have already solved, and I should use their solution.

            Show
            krzys Krzysztof Findeisen added a comment - Back In Progress, since Meredith Rawls pointed out that part of this is a problem the Gen 3 people have already solved, and I should use their solution.
            Hide
            krzys Krzysztof Findeisen added a comment - - edited

            Sorry Meredith Rawls, but could you re-review? Other than the corrections you requested, I think the only changes I made were to make_apdb.py (in particular, commit 057d803).

            Show
            krzys Krzysztof Findeisen added a comment - - edited Sorry Meredith Rawls , but could you re-review? Other than the corrections you requested, I think the only changes I made were to make_apdb.py (in particular, commit 057d803 ).
            Hide
            mrawls Meredith Rawls added a comment -

            I think this is good to go now, assuming Jenkins agrees. Thanks for your work on it!

            Show
            mrawls Meredith Rawls added a comment - I think this is good to go now, assuming Jenkins agrees. Thanks for your work on it!

              People

              Assignee:
              krzys Krzysztof Findeisen
              Reporter:
              krzys Krzysztof Findeisen
              Reviewers:
              Meredith Rawls
              Watchers:
              Chris Morrison, Krzysztof Findeisen, Meredith Rawls
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved: