Details
-
Type:
RFC
-
Status: Implemented
-
Resolution: Done
-
Component/s: DM
-
Labels:None
Description
After a year of development, we're finally ready to add the Alert Production pipeline to the Stack. The pipeline handles processing of raw data through image differencing and source association, producing a database of DIAObjects as its final output. It has been tested with both DECam and HSC data.
We'd like to add the following packages to top-level products:
- lsst_apps:
- ap_pipe: command-line task for running the entire AP pipeline. Depends on ap_association.
- ap_association: task for associating DIASources into DIAObjects. Depends on l1dbproto.
- l1dbproto: SLAC's current version of the prompt products database. As part of incorporation to the Stack this package should be renamed to follow Stack conventions (e.g., dax_ppdb?). Currently depends on db; this dependency is vestigial and should be removed.
- lsst_distrib:
- ap_verify: program for testing and monitoring the AP pipeline using lsst.verify. Depends on ap_pipe.
Note that many of these packages are in the lsst-dm GitHub organization, and would need to be moved to lsst. This RFC does not include adding the above packages' documentation to pipelines.lsst.io; I propose that this be done on a package-by-package basis at a later date.
Attachments
Issue Links
Activity
Sorry, I should have checked it earlier. There is no actual dependency on db in l1dbproto, I think I got rid of it when I switched to pex_config for configuration but forgot to update table file. Sorry for confusing everyone, the whole db direct and indirect dependencies should not have been there from beginning.
For reducing db dependencies (or making them optional) someone should open new ticket, I think this is relatively straightforward to implement.
If the dependency is illusory, then that's great news. Does this affect what you said on Sep. 12 about needing to split off some code and leave it in lsst-dm?
I'll modify the RFC proposal to remove the db dependency. Can you create the implementation ticket(s) for l1dbproto, since you have a clearer picture than I of what needs to be done?
Krzysztof Findeisen
added a comment - If the dependency is illusory, then that's great news. Does this affect what you said on Sep. 12 about needing to split off some code and leave it in lsst-dm ?
I'll modify the RFC proposal to remove the db dependency. Can you create the implementation ticket(s) for l1dbproto , since you have a clearer picture than I of what needs to be done? I still would like to split it and keep the code which is only used by my tests (this should be in bin/ folder) in lsst-dm. I'll make a ticket for this.
Since the only objection was to the low-level DB code, adopting as follows:
- l1dbproto will drop its db dependency, and will be mostly moved into lsst as described by Andy Salnikov
- the ap_* packages will be transferred to lsst wholesale
- these four packages will be added to lsst_apps or lsst_distrib as originally proposed
Krzysztof Findeisen
added a comment - Since the only objection was to the low-level DB code, adopting as follows:
l1dbproto will drop its db dependency, and will be mostly moved into lsst as described by Andy Salnikov
the ap_* packages will be transferred to lsst wholesale
these four packages will be added to lsst_apps or lsst_distrib as originally proposed
Decoupling dax_ppdb (or whatever we end up calling it) from db certainly seems like the best solution in the context of AP (though I also agree with Tim Jenness that db shouldn't require specific backends). Andy Salnikov, can we determine in advance whether the db dependency really can be removed?
(Also, why can it be removed? Is this a matter of duplicating code that's currently in db?)