Uploaded image for project: 'Request For Comments'
  1. Request For Comments
  2. RFC-587

Non-deprecated breaking change to AP pipeline workflow

    Details

    • Type: RFC
    • Status: Implemented
    • Resolution: Done
    • Component/s: DM
    • Labels:
      None

      Description

      Currently, ap_pipe.py tries to set up a prompt products database when it is run. This behavior is fine for Alert Production runs against an SQLite database on a single-node machine, but is expected to cause problems when moving to either a more formal (e.g., centrally managed) database system or a more complex computing environment. Therefore, DM-16606 adds a new script, make_ppdb.py, that creates a database for running ap_pipe.py in development environments. The intent is that, with an external script, ap_pipe.py can simply assume the database exists without worrying about whether it should (or can) set it up itself.

      The database creation script has been tested and reviewed, but removing the old code from ApPipeTask is a breaking change – users who previously could run ap_pipe.py with no preparation must ensure there's an available prompt products database*. We could have a deprecation period in which users are encouraged to call make_ppdb.py but ap_pipe.py also tries to set up the database (safe to try for SQLite), but that would limit our options for deploying the AP pipeline until the old code is finally removed.

      Since I understand that ap_pipe.py is starting to be used by other LSST developers, but hope the number of affected users is still low, I propose to not have a deprecation period and make ap_pipe.py require a pre-existing database as soon as DM-16606 is merged. Is there anyone who would be seriously affected by such a sudden change?

      [*Users of ap_verify.py don't need to do anything, only those who call ap_pipe.py directly.]

        Attachments

          Issue Links

            Activity

            Hide
            ktl Kian-Tat Lim added a comment -

            I'm OK with this; I don't think the original code is widely used, and the new flow seems better for the long term.

            Show
            ktl Kian-Tat Lim added a comment - I'm OK with this; I don't think the original code is widely used, and the new flow seems better for the long term.
            Hide
            swinbank John Swinbank added a comment -

            Bob Armstrong & Rahul Biswas — we identified you as the two most likely people to be impacted by this RFC. If you don't complain in the next day or so, we indent to adopt it (with DM-CCB approval), so please shout now if that's a problem for you.

            Show
            swinbank John Swinbank added a comment - Bob Armstrong & Rahul Biswas — we identified you as the two most likely people to be impacted by this RFC. If you don't complain in the next day or so, we indent to adopt it (with DM-CCB approval), so please shout now if that's a problem for you.
            Hide
            rearmstr Bob Armstrong added a comment -

            We are not currently using the ap_pipe implementation so this shouldn't affect us.

            Show
            rearmstr Bob Armstrong added a comment - We are not currently using the ap_pipe implementation so this shouldn't affect us.
            Hide
            swinbank John Swinbank added a comment -

            Per discussion at DMCCB#9, I have set this to “board recommended”. Krzysztof Findeisen, that means you can go ahead and adopt/implement this RFC at your leisure.

            Show
            swinbank John Swinbank added a comment - Per discussion at DMCCB#9 , I have set this to “board recommended”. Krzysztof Findeisen , that means you can go ahead and adopt/implement this RFC at your leisure.
            Hide
            krzys Krzysztof Findeisen added a comment -

            FYI, "Manager Approves" is not a very intuitive button for moving from "Board Recommended" to "Adopted"...

            Show
            krzys Krzysztof Findeisen added a comment - FYI, "Manager Approves" is not a very intuitive button for moving from "Board Recommended" to "Adopted"...

              People

              • Assignee:
                krzys Krzysztof Findeisen
                Reporter:
                krzys Krzysztof Findeisen
                Watchers:
                Bob Armstrong, John Parejko, John Swinbank, Kian-Tat Lim, Krzysztof Findeisen
              • Votes:
                0 Vote for this issue
                Watchers:
                5 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:
                  Planned End:

                  Summary Panel