Details
-
Type:
RFC
-
Status: Board Recommended
-
Resolution: Unresolved
-
Component/s: DM
-
Labels:None
Description
The Rubin Science Platform design, and the DM requirements, envision a number of user-facing services that perform computations, primarily on image data, in response to user requests. Examples:
- image cutouts
- image mosaics
- re-creation of non-persisted data products such as non-lossy-compressed PVIs and difference images
- creation of "virtual data products" - products such as special types of coadds that have a clear definition but are not actually produced or persisted in DRP
- forced photometry on demand
Image cutouts will be made available to users through IVOA-compliant SODA services. The other examples above do not correspond exactly to specific IVOA standards, but will be delivered through services compliant with lower-level IVOA standards for service implementation such as UWS (for asynchronous services), DALI, and VOSI.
The chosen architecture for deployment of such services envisions separating the mechanics of service implementation (parsing of URL APIs, generation of IVOA-style responses in XML, interface of UWS to project resources for running async jobs, etc.) from the afw-dependent implementation of the "back end" - the scientific content of the services.
This architecture permits the "back end" code to also be used directly in the Notebook Aspect, where desired, instead of accessing it as a service through a URL API.
Note that the back end code is doing things that are very similar to things done in the DRP (and, to some extent, AP) code base, but generally not in exactly the same way.
As an example: a PVI re-creation service will not be an exact duplicate of the original computation of PVIs from raw images, as it can rely on astrometric and PSF models that will have been persisted by the original production processing, requiring only the re-run of ISR and some new code to "stitch things back together" with the calibrations from the original processing.
It is highly desirable that this code be developed by the appropriate experts in the Science Pipelines organization, and subjected to comparable discipline in terms of CI and release procedures. The architecture supports a "separation of concerns" in that it separates code that requires knowledge of how to work with afw from code that has to obey IVOA standards and be suitable for use in persistent servers - corresponding to distinguishable areas of expertise in our individual developers and groups.
This RFC primarily addresses the question of where the back end code should go in the Rubin packaging system.
The Architecture team has looked at this question, with some input from others, and we are recommending the following:
- The existing lsst/dax/* namespace, which has been extensively hollowed out by the transition to TAP and other IVOA services in place of the original custom services, should be re-used to host the back-end implementations of the new services - that is, the afw-based code that does the actual computations. The namespace lsst/dax/image/ should be used for all the services mentioned in the list above*, and in particular the code from the proof-of-concept repository lsst-dm/image_cutouts_backend should be moved to lsst/dax/image/cutouts (i.e., lsst/dax_image_cutouts on GitHub).
- These backend packages (one now, with more planned) should be included in lsst-distrib and should be subjected to normal release discipline. This will help ensure that the backend implementations stay in sync with evolution in the underlying afw code and with the "pass one" implementations of these computations in DRP.
- To further support this aim, it may be appropriate to implement explicit tests for this, though - e.g., to include in the high-level tests for releases a test of the assertion that PVI-recreation yields the same results as the original processing.
- IVOA-service-layer code wrapping these backends will be delivered by the SQuaRE organization and will be in repositories in the lsst-sqre organization. No specific naming convention is recommended at present, but we encourage the SQuaRE team to adopt naming conventions that facilitate recognizing which repositories contain image service implementations. The service-layer code will not be in lsst-distrib and is not expected to be directly used by LSST science users (though of course remaining open-source as all of our code is).
Attachments
Issue Links
- mentioned in
-
Page Loading...
-
Page Loading...
-
Page Loading...
-
Page Loading...
-
Page Loading...
-
Page Loading...
-
Page Loading...
-
Page Loading...
-
Page Loading...
-
Page Loading...
-
Page Loading...
-
Page Loading...
-
Page Loading...
-
Page Loading...
-
Page Loading...
-
Page Loading...
-
Page Loading...
-
Page Loading...
-
Page Loading...
-
Page Loading...
-
Page Loading...
-
Page Loading...
-
Page Loading...
-
Page Loading...
-
Page Loading...
-
Page Loading...
-
Page Loading...
-
Page Loading...
-
Page Loading...
-
Page Loading...
-
Page Loading...
-
Page Loading...
-
Page Loading...
-
Page Loading...
-
Page Loading...
-
Page Loading...
-
Page Loading...
-
Page Loading...
-
Page Loading...
-
Page Loading...
-
Page Loading...
-
Page Loading...
-
Page Loading...
-
Page Loading...
-
Page Loading...
-
Page Loading...
-
Page Loading...
-
Page Loading...
-
Page Loading...
-
Page Loading...
-
Page Loading...
-
Page Loading...
-
Page Loading...
-
Page Loading...
-
Page Loading...
-
Page Loading...
-
Page Loading...
-
Page Loading...
-
Page Loading...
-
Page Loading...
-
Page Loading...
-
Page Loading...
The frontend code does not use a Science Pipelines container because there's no need for the stack for the frontend (and in fact the stack is actively undesirable because it pins a bunch of Python libraries to versions that may not be the ones we use for the frontend). Since the frontend is a simple Python service, it is built on a generic Python container.
The frontend communicates with the backend via a queue and they run in separate pods. See the design at https://dmtn-208.lsst.io/#architecture-summary