Uploaded image for project: 'Request For Comments'
  1. Request For Comments
  2. RFC-521

Add ap_verify and its dependencies to the Stack

    Details

    • Type: RFC
    • Status: Implemented
    • Resolution: Done
    • Component/s: DM
    • Labels:
      None

      Description

      After a year of development, we're finally ready to add the Alert Production pipeline to the Stack. The pipeline handles processing of raw data through image differencing and source association, producing a database of DIAObjects as its final output. It has been tested with both DECam and HSC data.

      We'd like to add the following packages to top-level products:

      • lsst_apps:
        • ap_pipe: command-line task for running the entire AP pipeline. Depends on ap_association.
        • ap_association: task for associating DIASources into DIAObjects. Depends on l1dbproto.
        • l1dbproto: SLAC's current version of the prompt products database. As part of incorporation to the Stack this package should be renamed to follow Stack conventions (e.g., dax_ppdb?). Currently depends on db; this dependency is vestigial and should be removed.
      • lsst_distrib:
        • ap_verify: program for testing and monitoring the AP pipeline using lsst.verify. Depends on ap_pipe.

      Note that many of these packages are in the lsst-dm GitHub organization, and would need to be moved to lsst. This RFC does not include adding the above packages' documentation to pipelines.lsst.io; I propose that this be done on a package-by-package basis at a later date.

        Attachments

          Issue Links

            Activity

            Hide
            krzys Krzysztof Findeisen added a comment -

            Decoupling dax_ppdb (or whatever we end up calling it) from db certainly seems like the best solution in the context of AP (though I also agree with Tim Jenness that db shouldn't require specific backends). Andy Salnikov, can we determine in advance whether the db dependency really can be removed?

            (Also, why can it be removed? Is this a matter of duplicating code that's currently in db?)

            Show
            krzys Krzysztof Findeisen added a comment - Decoupling dax_ppdb (or whatever we end up calling it) from db certainly seems like the best solution in the context of AP (though I also agree with Tim Jenness that db shouldn't require specific backends). Andy Salnikov , can we determine in advance whether the db dependency really can be removed? (Also, why can it be removed? Is this a matter of duplicating code that's currently in db ?)
            Hide
            salnikov Andy Salnikov added a comment -

            Sorry, I should have checked it earlier. There is no actual dependency on db in l1dbproto, I think I got rid of it when I switched to pex_config for configuration but forgot to update table file. Sorry for confusing everyone, the whole db direct and indirect dependencies should not have been there from beginning.

            For reducing db dependencies (or making them optional) someone should open new ticket, I think this is relatively straightforward to implement.

            Show
            salnikov Andy Salnikov added a comment - Sorry, I should have checked it earlier. There is no actual dependency on db in l1dbproto , I think I got rid of it when I switched to pex_config for configuration but forgot to update table file. Sorry for confusing everyone, the whole db direct and indirect dependencies should not have been there from beginning. For reducing db dependencies (or making them optional) someone should open new ticket, I think this is relatively straightforward to implement.
            Hide
            krzys Krzysztof Findeisen added a comment -

            If the dependency is illusory, then that's great news. Does this affect what you said on Sep. 12 about needing to split off some code and leave it in lsst-dm?

            I'll modify the RFC proposal to remove the db dependency. Can you create the implementation ticket(s) for l1dbproto, since you have a clearer picture than I of what needs to be done?

            Show
            krzys Krzysztof Findeisen added a comment - If the dependency is illusory, then that's great news. Does this affect what you said on Sep. 12 about needing to split off some code and leave it in lsst-dm ? I'll modify the RFC proposal to remove the db dependency. Can you create the implementation ticket(s) for l1dbproto , since you have a clearer picture than I of what needs to be done?
            Hide
            salnikov Andy Salnikov added a comment -

            I still would like to split it and keep the code which is only used by my tests (this should be in bin/ folder) in lsst-dm. I'll make a ticket for this.

            Show
            salnikov Andy Salnikov added a comment - I still would like to split it and keep the code which is only used by my tests (this should be in bin/ folder) in lsst-dm . I'll make a ticket for this.
            Hide
            krzys Krzysztof Findeisen added a comment -

            Since the only objection was to the low-level DB code, adopting as follows:

            • l1dbproto will drop its db dependency, and will be mostly moved into lsst as described by Andy Salnikov
            • the ap_* packages will be transferred to lsst wholesale
            • these four packages will be added to lsst_apps or lsst_distrib as originally proposed
            Show
            krzys Krzysztof Findeisen added a comment - Since the only objection was to the low-level DB code, adopting as follows: l1dbproto will drop its db dependency, and will be mostly moved into lsst as described by Andy Salnikov the ap_* packages will be transferred to lsst wholesale these four packages will be added to lsst_apps or lsst_distrib as originally proposed

              People

              • Assignee:
                krzys Krzysztof Findeisen
                Reporter:
                krzys Krzysztof Findeisen
                Watchers:
                Andy Salnikov, Chris Morrison, Eric Bellm, Fritz Mueller, John Parejko, John Swinbank, Krzysztof Findeisen, Meredith Rawls, Simon Krughoff, Tim Jenness
              • Votes:
                0 Vote for this issue
                Watchers:
                10 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:
                  Planned End:

                  Summary Panel