Uploaded image for project: 'Request For Comments'
  1. Request For Comments
  2. RFC-325

Investigate Storing Processed Visit Images (PVIs) and Making PVIs Available to Users


    • Type: RFC
    • Status: Adopted
    • Resolution: Unresolved
    • Component/s: DM
    • Labels:


      This RFC is a proposal to keep the Processed Visit Images (PVIs) as part of the Level 1 and Level 2 data processing and store the PVIs in a manner readily accessible to the Users through the Science Platform.

      October 2017 Action

      As of October 2017, discussion of this RFC has resulting in the following framework for proposed changes to the baseline design:

      • Maintain uncompressed or lossless-compressed copies of the per-exposure raw data on spinning disk at both the U.S. and Chilean LSST data centers. The detailed technical implications and costs of this will now be studied (DM-11880).
      • Study the feasibility of maintaining lossy-compressed copies of the PVIs on spinning disk. The degree of compression and the scientific trade-offs involved will be studied, largely within a newly commissioned Data Compression Working Group (DM-11819). With lossy-compressed PVIs available, a low-latency PVI service can be offered, with a second layer of a higher-latency on-demand PVI-recalculation service for scientifically demanding use cases. Appropriate latency requirements for these services will be studied (DM-11879).
      • Continue to explore the options for coordinating the DM PVI service(s) with the EPO group's desire for the ability to display time-dependent imaging data to its users (DM-11881).

      Adoption of the present RFC signals the authorization of the work required to support these studies. The studies in these three areas will be carried out over the next few months and will result in, at first, a new RFC presenting a new design proposal and the associated technical and cost impacts, and then a formal LSST-level change request that inserts the changes into high-level requirements and design documents, and to the PMCS.

      Scientific Rationale for Maintaining the Processed Visit Images (PVIs) and having the PVIs available as product in (near) real time to the users.

      Each of the four pillars of the LSST science cases rely on both the time-averaged co-addition images and the single processed visit images (i.e., time dependent data). Currently, the Processed Visit Images (PVIs) are not stored as part of the available data products, but rather the baseline model is to produce the PVIs on-demand as the users need them. However, the latency of producing the PVIs on-the-fly will significantly hamper the ability of the users to display PVIs in a reasonable time and to do scientific inquiries in support of the four LSST science pillars. Further, it is possible that there is not enough computing resources with the LSST DAC to produce the PVIs on-demand, given the scale of the LSST data.

      For time-dependent studies having 'near-immediate' access to the PVIs is critical to evaluation of the quality and cause of the associated alerts and the time-series data. Time series studies inherently are about searching for things that change either in predictable (e.g., periodic variables or moving objects) or unpredictable (e.g., eruptive variables, moving objects) ways.

      Full evaluation of the photometric time series of an Object requires having access to the imaging time series that corresponds to the Source photometry. By enabling users to quickly access and visualize the images from which the Source photometry are drawn, users will be able to assess the quality and "reality" of the photometric variations. Without access to the images, full assessment of the quality of the photometric properties as a function of time will not be possible. In particular, for Objects that demonstrate unexpected brightness changes (either drop-outs or brightenings), evaluation of the images that correspond to the time series is critical to the scientific assessment of the Object.


      Near-Earth Objects

      • Moving object appears to brighten suddenly and then dims. If the brightening is real, it could be the result of the Object is asymmetric and is tumbling such that its largest face is now towards the Earth. Or it is possible that the brightening of the object is caused the Object passing in front of a background star or galaxy, and the photometric deblending algorithm was compromised by the presence of the moving object. Direct examination of the images can be used to assess this.

      Eruptive Variables

      • Often eruptive variables show signs of low-level variability years prior to the full eruptive event (e.g., Supernovae, dwarf novae, FU Ori-stars). At the precision and sensitivity of LSST, assessing the quality of potential pre-cursor variability requires access to and evalutation of the time series images associated with the Objects.

      Changes in Periodic Variables:

      • Often periodic variables (e.g., eclipsing binaries, RR Lyrae, Cepheids) show changes in the expected photometric behavior sometimes caused by the presence of unknown and unresolved additional objects in the system. Evalution of the quality of the unexpected brightness change to ensure that, for example, the result was not from a passing moving object or from changes in the blending of nearby objects requires examination of the images that produced the photometric variations.

      As the survey progresses, Objects may be observed approximately a few to hundreds of times, depending on its location on the sky. Further, Solar System Objects move and may be observed a few to hundreds of times depending on the location of the Object within the Solar System. The survey will generate observations of billions of non-solar Objects and millions of Solar System Objects - all of which will be variable (in brightness and/or position) to some degree to within the photometric precision of LSST. The user community will potentially generate millions to billions of requests to generate image cutouts from the PVIs, depending on whether the users wish to see a single time-stamped view or the entire time series of images.

      Generation of the PVIs on the fly could potentially overwhelm the system or, if not available to the users, significantly hamper the ability of the users to do scientific evaluations of the time series aspect of the LSST data.

      This has implications for the sizing requirements of the DAC; some of that sizing modeling work has been started at


          Issue Links



              • Assignee:
                gpdf Gregory Dubois-Felsmann
                Ciardi David Ciardi [X] (Inactive)
                Ben Emmons [X] (Inactive), David Ciardi [X] (Inactive), Donald Petravick, Frossie Economou, Gregory Dubois-Felsmann, Hsin-Fang Chiang, John Swinbank, Kian-Tat Lim, Leanne Guy, Margaret Gelman, Tim Jenness, Wil O'Mullane, Xiuqin Wu [X] (Inactive), Zeljko Ivezic
              • Votes:
                1 Vote for this issue
                14 Start watching this issue


                • Created:
                  Planned End:

                  Summary Panel