Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-14067

Incomplete flowdown of OSS provenance requirements to DMSR

    XMLWordPrintable

Details

    • Arch 2018-09-24, Arch 2018-10-08, Arch 2018-10-15, Arch 2018-10-22, Arch 2018-10-29, Arch 2018-11-05, Arch 2018-11-12, Arch 2018-11-19, Arch 2018-11-26
    • Architecture

    Description

      In reviewing the flowdown for provenance requirements on SuperTask and the Butler-related middleware, I noticed that the high-level requirements OSS-REQ-0122, Provenance:

      The LSST Data Management system shall record provenance data on all its processing activities: all information necessary to reproduce computed data products from the associated raw data, and to understand the processing history of any data product.

      This shall include at least: software version and build information, settings of all configurable parameters, history of processing steps, identification of all calibration constants used in processing, and hardware and operating system configurations used.

      and OSS-REQ-0172, Provenance Archiving:

      The LSST project shall archive all processing provenance associated with archived data products.

      were never properly flowed down to the DMSR. There are numerous DMSR requirements that mention provenance - regarding maintaining it for specific data products, such as coadds (DMS-REQ-0106), or querying against it, and one that mandates a generic provenance mechanism usable in "Level 3" analyses. But there are no explicit functional requirements at DMSR on the creation of the mechanisms or their features.

      This makes it hard to verify their flowdown to the middleware requirements, LDM-556, at which level we currently cover some, but not all, of the features required to meet the OSS requirements.

      Attachments

        Issue Links

          Activity

            It looks like we have also failed to flow down the Reproducibility requirement, OSS-REQ-0123:

            Specification: The LSST data management system shall ensure that the results of processing of data are reproducible. Any data processing task, when re-run based on the provenance data from the previous run, on the same system, shall produce the same results (with the exception of provenance data or other execution records that depend on the wall-clock time or on variable system loads). Tasks re-run on different systems shall produce the same results to the extent computationally feasible.

            Discussion: "Computationally feasible" refers to the fact that floating-point operations typically can return slightly different results on different hardware platforms. LSST code is required to use reasonable care in the implementation of floating-point computations to avoid the unnecessary accumulation of error, but is not required to adopt computationally costly defensive techniques to avoid differences altogether.

            gpdf Gregory Dubois-Felsmann added a comment - It looks like we have also failed to flow down the Reproducibility requirement, OSS-REQ-0123: Specification: The LSST data management system shall ensure that the results of processing of data are reproducible. Any data processing task, when re-run based on the provenance data from the previous run, on the same system, shall produce the same results (with the exception of provenance data or other execution records that depend on the wall-clock time or on variable system loads). Tasks re-run on different systems shall produce the same results to the extent computationally feasible. Discussion: "Computationally feasible" refers to the fact that floating-point operations typically can return slightly different results on different hardware platforms. LSST code is required to use reasonable care in the implementation of floating-point computations to avoid the unnecessary accumulation of error, but is not required to adopt computationally costly defensive techniques to avoid differences altogether.
            gcomoretto Gabriele Comoretto [X] (Inactive) added a comment - LCR-1465  https://project.lsst.org/groups/ccb/node/2681 opened  

            I've made some edits on the LCR after discussion in the Systems Engineering group about "multiple shalls" and the like.

            gpdf Gregory Dubois-Felsmann added a comment - I've made some edits on the LCR after discussion in the Systems Engineering group about "multiple shalls" and the like.

            LCR-1465 Fully Implemented

            gcomoretto Gabriele Comoretto [X] (Inactive) added a comment - LCR-1465 Fully Implemented

            People

              gcomoretto Gabriele Comoretto [X] (Inactive)
              gpdf Gregory Dubois-Felsmann
              Brian Van Klaveren, Gabriele Comoretto [X] (Inactive), Gregory Dubois-Felsmann, Kian-Tat Lim, Leanne Guy, Tim Jenness
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Jenkins

                  No builds found.