Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-21886

Create PipelineTask driver for ap_pipe tasks that interact with the APDB

    Details

    • Story Points:
      8
    • Sprint:
      AP F19-6 (November), AP S20-1 (December)
    • Team:
      Alert Production

      Description

      Create a PipelineTask whose runQuantum method includes all of the logic in ApPipeTask.runAssociation.

      This should probably be done after DM-21877, so it can use the same approach to signal completion of APDB writes to downstream PipelineTasks.

      It's not clear what to do about ownership of ApPipeTask.ppdb. It must belong to the new task in Gen 3, but doing so will break config overrides in Gen 2 (and the DB location is always overridden).

        Attachments

          Issue Links

            Activity

            Show
            cmorrison Chris Morrison added a comment - New Jenkins:  https://ci.lsst.codes/blue/organizations/jenkins/stack-os-matrix/detail/stack-os-matrix/30932/pipeline
            Hide
            krzys Krzysztof Findeisen added a comment -

            Fine by me, as long as it's a blocker for DM-21939.

            Show
            krzys Krzysztof Findeisen added a comment - Fine by me, as long as it's a blocker for DM-21939 .
            Hide
            cmorrison Chris Morrison added a comment -

            Why don't I make a ticket to make DiaPipelineTask a subtask? There are a lot of specific calls to the associator and other subtasks throughout the code and more than a few settings in the Apdb that need to be taken care of here and in ap_verify. Seems like a good idea to separate that work a little.

            Show
            cmorrison Chris Morrison added a comment - Why don't I make a ticket to make DiaPipelineTask a subtask? There are a lot of specific calls to the associator and other subtasks throughout the code and more than a few settings in the Apdb that need to be taken care of here and in ap_verify. Seems like a good idea to separate that work a little.
            Hide
            krzys Krzysztof Findeisen added a comment -

            I did miss the new DiaPipelineTask, sorry about that.

            Given that it exists, I guess my question is why you can't make it a subtask of the Gen 2 ApPipeTask and call it from runAssociation (all the inputs for DiaPipelineTask.run correspond to sensorRef.get calls in the old code). That would avoid the current code duplication and make the remaining migration steps much clearer. No matter what we do, ApPipeTask and the new yaml file will have to coexist for a bit (up to mid-2020, as of this week's middleware demo), so code duplication could be a real problem.

            If you're not going to make DiaPipelineTask a subtask of ApPipeTask, please add a comment about it to DM-21939, since it's not very discoverable at the moment.

            Show
            krzys Krzysztof Findeisen added a comment - I did miss the new DiaPipelineTask , sorry about that. Given that it exists, I guess my question is why you can't make it a subtask of the Gen 2 ApPipeTask and call it from runAssociation (all the inputs for DiaPipelineTask.run correspond to sensorRef.get calls in the old code). That would avoid the current code duplication and make the remaining migration steps much clearer. No matter what we do, ApPipeTask and the new yaml file will have to coexist for a bit (up to mid-2020, as of this week's middleware demo), so code duplication could be a real problem. If you're not going to make DiaPipelineTask a subtask of ApPipeTask , please add a comment about it to DM-21939 , since it's not very discoverable at the moment.
            Hide
            cmorrison Chris Morrison added a comment -

            Hey Krzysztof Findeisen, did you checkout the new DiaPipelineTask method that comes along with this ticket? That's what's going to be "runAssociation" in the future. I kept ap_pipe as is (and slightly modified) so that it still works with Gen2 for now. I'm assuming that once we are ready, ap_pipe will just become a yaml file that will have DiaPipelineTask as one if it's pipelines or am I miss reading what is happening?

            Show
            cmorrison Chris Morrison added a comment - Hey Krzysztof Findeisen , did you checkout the new DiaPipelineTask method that comes along with this ticket? That's what's going to be "runAssociation" in the future. I kept ap_pipe as is (and slightly modified) so that it still works with Gen2 for now. I'm assuming that once we are ready, ap_pipe will just become a yaml file that will have DiaPipelineTask as one if it's pipelines or am I miss reading what is happening?

              People

              • Assignee:
                cmorrison Chris Morrison
                Reporter:
                jbosch Jim Bosch
                Reviewers:
                Ian Sullivan
                Watchers:
                Chris Morrison, Eric Bellm, Ian Sullivan, Jim Bosch, Krzysztof Findeisen
              • Votes:
                0 Vote for this issue
                Watchers:
                5 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Summary Panel