Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-21939

Create Gen 3 AP Pipeline

    Details

    • Story Points:
      4
    • Sprint:
      AP F19-6 (November), AP S20-6 (May), AP F20-1 (June)
    • Team:
      Alert Production

      Description

      This is a mostly-umbrella ticket for getting all current Gen2 ap_pipe functionality working in Gen3. The goal is being able to run AP on HiTS 2015 using the CmdLineActivator. It includes:

      • creating scripts that ingest HiTS test data into a Gen3 repository (DM-21862)
      • conversion of all remaining ApPipeTask subtasks to Gen 3 (DM-21874, DM-21886)
      • creation of a YAML file configuring the pipeline (1 SP on this ticket)

      ApPipeTask has a lot of code for handling the case of calexp difference imaging templates. By general agreement within the AP and Middleware groups, we will not be porting this functionality to Gen 3 (see also DM-21874), and will reimplement it from scratch should we need it after migration.

        Attachments

        1. ap_pipe_docs.tar.gz
          210 kB
          Krzysztof Findeisen
        2. convertRepo_standalone.py
          0.3 kB
          Krzysztof Findeisen

          Issue Links

            Activity

            Hide
            mrawls Meredith Rawls added a comment -

            I left straightforward comments on GitHub for the ap_pipe docs, and pending those, it looks good. I'll take the liberty of marking this as "reviewed" for all three of us.

            One somewhat larger question - is the Gen 3 implementation done here kind of an intermediate step while we still need to support Gen 2? It's my vague understanding that eventually, running the AP Pipeline will just be having a yaml file with "do ISR, do characterization, do calibration, do differencing, do association, the end" and we won't need an ap_pipe package to link those steps together. Maybe something to discuss at our weekly meeting if I'm off base. For now, I'm thrilled it works!

            Show
            mrawls Meredith Rawls added a comment - I left straightforward comments on GitHub for the ap_pipe docs, and pending those, it looks good. I'll take the liberty of marking this as "reviewed" for all three of us. One somewhat larger question - is the Gen 3 implementation done here kind of an intermediate step while we still need to support Gen 2? It's my vague understanding that eventually, running the AP Pipeline will just be having a yaml file with "do ISR, do characterization, do calibration, do differencing, do association, the end" and we won't need an ap_pipe package to link those steps together. Maybe something to discuss at our weekly meeting if I'm off base. For now, I'm thrilled it works!
            Hide
            krzys Krzysztof Findeisen added a comment - - edited

            What about self.metadata? What was the reason for the introduction of algMetadata?

            I don't know anything about algMetadata, including whether or not it's related to metadata (though its attachment to source catalogs would make me think it's something different). Since Gen 3 tasks are supposed to be stateless, I assume metadata is going away at some point, but last I checked Middleware didn't have a plan for a replacement yet.

            Show
            krzys Krzysztof Findeisen added a comment - - edited What about self.metadata ? What was the reason for the introduction of algMetadata ? I don't know anything about algMetadata , including whether or not it's related to metadata (though its attachment to source catalogs would make me think it's something different). Since Gen 3 tasks are supposed to be stateless, I assume metadata is going away at some point, but last I checked Middleware didn't have a plan for a replacement yet.
            Hide
            krzys Krzysztof Findeisen added a comment - - edited

            It's my vague understanding that eventually, running the AP Pipeline will just be having a yaml file with "do ISR, do characterization, do calibration, do differencing, do association, the end" and we won't need an ap_pipe package to link those steps together

            I think as long as we have ApPipeTask, it would be confusing for users to need different packages depending on whether they're running the pipeline in Gen 2 or Gen 3. So I'd be opposed to moving to pipe_tasks before we phase out ApPipeTask. We'd also need to do something about make_apdb or its replacement (see DM-22663); maybe put it in ap_association?

            Show
            krzys Krzysztof Findeisen added a comment - - edited It's my vague understanding that eventually, running the AP Pipeline will just be having a yaml file with "do ISR, do characterization, do calibration, do differencing, do association, the end" and we won't need an ap_pipe package to link those steps together I think as long as we have ApPipeTask , it would be confusing for users to need different packages depending on whether they're running the pipeline in Gen 2 or Gen 3. So I'd be opposed to moving to pipe_tasks before we phase out ApPipeTask . We'd also need to do something about make_apdb or its replacement (see DM-22663 ); maybe put it in ap_association ?
            Hide
            tjenness Tim Jenness added a comment -

            What is this metadata that is being discussed?

            Show
            tjenness Tim Jenness added a comment - What is this metadata that is being discussed?
            Hide
            krzys Krzysztof Findeisen added a comment - - edited

            One is the standard Task.metadata. The other is ImageDifferenceTask.algMetadata, which seems to be associated with measurement algorithms in some way.

            Show
            krzys Krzysztof Findeisen added a comment - - edited One is the standard Task.metadata . The other is ImageDifferenceTask.algMetadata , which seems to be associated with measurement algorithms in some way.

              People

              • Assignee:
                krzys Krzysztof Findeisen
                Reporter:
                krzys Krzysztof Findeisen
                Reviewers:
                Chris Morrison, Gabor Kovacs, Meredith Rawls
                Watchers:
                Chris Morrison, Gabor Kovacs, John Swinbank, Krzysztof Findeisen, Meredith Rawls, Tim Jenness
              • Votes:
                0 Vote for this issue
                Watchers:
                6 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Summary Panel