Uploaded image for project: 'Request For Comments'
  1. Request For Comments
  2. RFC-325

Investigate Storing Processed Visit Images (PVIs) and Making PVIs Available to Users

    Details

    • Type: RFC
    • Status: Adopted
    • Resolution: Unresolved
    • Component/s: DM
    • Labels:

      Description

      This RFC is a proposal to keep the Processed Visit Images (PVIs) as part of the Level 1 and Level 2 data processing and store the PVIs in a manner readily accessible to the Users through the Science Platform.

      October 2017 Action

      As of October 2017, discussion of this RFC has resulting in the following framework for proposed changes to the baseline design:

      • Maintain uncompressed or lossless-compressed copies of the per-exposure raw data on spinning disk at both the U.S. and Chilean LSST data centers. The detailed technical implications and costs of this will now be studied (DM-11880).
      • Study the feasibility of maintaining lossy-compressed copies of the PVIs on spinning disk. The degree of compression and the scientific trade-offs involved will be studied, largely within a newly commissioned Data Compression Working Group (DM-11819). With lossy-compressed PVIs available, a low-latency PVI service can be offered, with a second layer of a higher-latency on-demand PVI-recalculation service for scientifically demanding use cases. Appropriate latency requirements for these services will be studied (DM-11879).
      • Continue to explore the options for coordinating the DM PVI service(s) with the EPO group's desire for the ability to display time-dependent imaging data to its users (DM-11881).

      Adoption of the present RFC signals the authorization of the work required to support these studies. The studies in these three areas will be carried out over the next few months and will result in, at first, a new RFC presenting a new design proposal and the associated technical and cost impacts, and then a formal LSST-level change request that inserts the changes into high-level requirements and design documents, and to the PMCS.

      Scientific Rationale for Maintaining the Processed Visit Images (PVIs) and having the PVIs available as product in (near) real time to the users.

      Each of the four pillars of the LSST science cases rely on both the time-averaged co-addition images and the single processed visit images (i.e., time dependent data). Currently, the Processed Visit Images (PVIs) are not stored as part of the available data products, but rather the baseline model is to produce the PVIs on-demand as the users need them. However, the latency of producing the PVIs on-the-fly will significantly hamper the ability of the users to display PVIs in a reasonable time and to do scientific inquiries in support of the four LSST science pillars. Further, it is possible that there is not enough computing resources with the LSST DAC to produce the PVIs on-demand, given the scale of the LSST data.

      For time-dependent studies having 'near-immediate' access to the PVIs is critical to evaluation of the quality and cause of the associated alerts and the time-series data. Time series studies inherently are about searching for things that change either in predictable (e.g., periodic variables or moving objects) or unpredictable (e.g., eruptive variables, moving objects) ways.

      Full evaluation of the photometric time series of an Object requires having access to the imaging time series that corresponds to the Source photometry. By enabling users to quickly access and visualize the images from which the Source photometry are drawn, users will be able to assess the quality and "reality" of the photometric variations. Without access to the images, full assessment of the quality of the photometric properties as a function of time will not be possible. In particular, for Objects that demonstrate unexpected brightness changes (either drop-outs or brightenings), evaluation of the images that correspond to the time series is critical to the scientific assessment of the Object.

      Examples:

      Near-Earth Objects

      • Moving object appears to brighten suddenly and then dims. If the brightening is real, it could be the result of the Object is asymmetric and is tumbling such that its largest face is now towards the Earth. Or it is possible that the brightening of the object is caused the Object passing in front of a background star or galaxy, and the photometric deblending algorithm was compromised by the presence of the moving object. Direct examination of the images can be used to assess this.

      Eruptive Variables

      • Often eruptive variables show signs of low-level variability years prior to the full eruptive event (e.g., Supernovae, dwarf novae, FU Ori-stars). At the precision and sensitivity of LSST, assessing the quality of potential pre-cursor variability requires access to and evalutation of the time series images associated with the Objects.

      Changes in Periodic Variables:

      • Often periodic variables (e.g., eclipsing binaries, RR Lyrae, Cepheids) show changes in the expected photometric behavior sometimes caused by the presence of unknown and unresolved additional objects in the system. Evalution of the quality of the unexpected brightness change to ensure that, for example, the result was not from a passing moving object or from changes in the blending of nearby objects requires examination of the images that produced the photometric variations.

      As the survey progresses, Objects may be observed approximately a few to hundreds of times, depending on its location on the sky. Further, Solar System Objects move and may be observed a few to hundreds of times depending on the location of the Object within the Solar System. The survey will generate observations of billions of non-solar Objects and millions of Solar System Objects - all of which will be variable (in brightness and/or position) to some degree to within the photometric precision of LSST. The user community will potentially generate millions to billions of requests to generate image cutouts from the PVIs, depending on whether the users wish to see a single time-stamped view or the entire time series of images.

      Generation of the PVIs on the fly could potentially overwhelm the system or, if not available to the users, significantly hamper the ability of the users to do scientific evaluations of the time series aspect of the LSST data.

      This has implications for the sizing requirements of the DAC; some of that sizing modeling work has been started at
      https://confluence.lsstcorp.org/display/DM/Notes+on+the+FDR-era+Sizing+Model

        Attachments

          Issue Links

            Activity

            Hide
            tjenness Tim Jenness added a comment -

            We adopted this RFC as an investigative RFC (I should change the title). Once the triggered tickets linked from this RFC are resolved we will have enough information for an LCR to be submitted.

            Show
            tjenness Tim Jenness added a comment - We adopted this RFC as an investigative RFC (I should change the title). Once the triggered tickets linked from this RFC are resolved we will have enough information for an LCR to be submitted.
            Hide
            ktl Kian-Tat Lim added a comment -

            Using a figure of 4.5X from the lossy compression Working Group, I have updated David Ciardi's slides with the estimated cost for that option.

            RFC-325summary_20180522_ktl.pptx

            Show
            ktl Kian-Tat Lim added a comment - Using a figure of 4.5X from the lossy compression Working Group, I have updated David Ciardi's slides with the estimated cost for that option. RFC-325summary_20180522_ktl.pptx
            Hide
            gpdf Gregory Dubois-Felsmann added a comment -

            It was noted today at the DM-SST meeting that the design and cost impact research needed to permit an LCR that implements the notional plan from this RFC is still pending.

            Show
            gpdf Gregory Dubois-Felsmann added a comment - It was noted today at the DM-SST meeting that the design and cost impact research needed to permit an LCR that implements the notional plan from this RFC is still pending.
            Hide
            frossie Frossie Economou added a comment -

            Can someone point me to the LCR in question?

            Show
            frossie Frossie Economou added a comment - Can someone point me to the LCR in question?
            Hide
            tjenness Tim Jenness added a comment - - edited

            This is a spinoff of LCR-908, I don't think we have yet made a formal LCR for the PVIs.

            Show
            tjenness Tim Jenness added a comment - - edited This is a spinoff of LCR-908 , I don't think we have yet made a formal LCR for the PVIs.

              People

              • Assignee:
                gpdf Gregory Dubois-Felsmann
                Reporter:
                Ciardi David Ciardi [X] (Inactive)
                Watchers:
                Ben Emmons [X] (Inactive), David Ciardi [X] (Inactive), Donald Petravick, Frossie Economou, Gregory Dubois-Felsmann, Hsin-Fang Chiang, John Swinbank, Kian-Tat Lim, Leanne Guy, Margaret Gelman, Tim Jenness, Wil O'Mullane, Xiuqin Wu [X] (Inactive), Zeljko Ivezic
              • Votes:
                1 Vote for this issue
                Watchers:
                14 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Planned End:

                  Summary Panel