Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-13190

Propose DM text for LCR-786 - elimination of Base raw data network outage buffer

    Details

    • Team:
      Architecture

      Description

      Following up on a discussion at the project SE-IPT meeting today, the action here is to propose specific changes to the OSS and DMSR that capture the intent of the change in LCR-786, the elimination of the Base network outage buffer and the increase in size of the Camera DAQ data buffer.

      Proposed language is meant to be introduced at the Systems Engineering meeting on January 30th, 2018.

        Attachments

          Issue Links

            Activity

            Hide
            tjenness Tim Jenness added a comment -

            LCR-786 is vague concerning its effect on DM (other than the addition of $40k to DM budget KLM20602A.PP to pay for the summit disks).

            The vote result stated:

            DM will remove the Base "network outage buffer" from the design of the data-handling system;

            Previously 14 days was mentioned as being removed from the base and that suggests we have to do something about DMS-REQ-0177:

            Base Facility Temporary Storage

            Specification: The Base Facility shall provide at least a time minBaseDataStorage (or half of that amount, redundantly) of raw data storage in the event of Base to Archive Center network outage.

            where minBaseDataStorage is 14 days, and that requirement is associated with DMS-REQ-0164 (Temporary Storage for Communications Links) and OSS-REQ-0054 (Base Data Buffer):

            Specification: The LSST system shall provide a limited buffer for data (raw, processed, and engineering)
            en route from the Base to the Archive Center, to allow science operations to proceed in the event of
            communications loss between the Base and Archive. The system shall have sufficient capacity to store
            all en route data for at least the design outage time, baseConnectivityLossTime [48 hours].

            It was noted by Kian-Tat Lim that DMS-REQ-0177 might still be needed because some sort of buffer will be used for other parts of the system (but 0177 doesn't really say what the size should be).

            Which design documents referred to in the LCR discuss the base buffer? Are those all now fixed as part of the re-plan?

            Show
            tjenness Tim Jenness added a comment - LCR-786 is vague concerning its effect on DM (other than the addition of $40k to DM budget KLM20602A.PP to pay for the summit disks). The vote result stated: DM will remove the Base "network outage buffer" from the design of the data-handling system; Previously 14 days was mentioned as being removed from the base and that suggests we have to do something about DMS-REQ-0177: Base Facility Temporary Storage Specification: The Base Facility shall provide at least a time minBaseDataStorage (or half of that amount, redundantly) of raw data storage in the event of Base to Archive Center network outage. where minBaseDataStorage is 14 days, and that requirement is associated with DMS-REQ-0164 (Temporary Storage for Communications Links) and OSS-REQ-0054 (Base Data Buffer): Specification: The LSST system shall provide a limited buffer for data (raw, processed, and engineering) en route from the Base to the Archive Center, to allow science operations to proceed in the event of communications loss between the Base and Archive. The system shall have sufficient capacity to store all en route data for at least the design outage time, baseConnectivityLossTime [48 hours] . It was noted by Kian-Tat Lim that DMS-REQ-0177 might still be needed because some sort of buffer will be used for other parts of the system (but 0177 doesn't really say what the size should be). Which design documents referred to in the LCR discuss the base buffer? Are those all now fixed as part of the re-plan?
            Hide
            tjenness Tim Jenness added a comment - - edited

            After discussion with Kian-Tat Lim on Slack we think that a reasonable approach is:

            1. Remove DMS-REQ-0177
            2. DMS-REQ-0164 be adjusted to end with "at or before the source end of that link."
            3. DMS-REQ-0318: Changing 3-day in discussion section to summit buffer size + 1 day.
            Show
            tjenness Tim Jenness added a comment - - edited After discussion with Kian-Tat Lim on Slack we think that a reasonable approach is: Remove DMS-REQ-0177 DMS-REQ-0164 be adjusted to end with "at or before the source end of that link." DMS-REQ-0318: Changing 3-day in discussion section to summit buffer size + 1 day.
            Hide
            ktl Kian-Tat Lim added a comment -

            For the OSS, my proposal is:

            1. OSS-REQ-0004: change "reprocessing" to "processing" (leftover from AP move).
            2. OSS-REQ-0052: summitBufferTransferTime should be rewritten as a transfer rate (2 days per day).  Note that DMS-REQ-0165 catches up processing at one day per day, but transfer can and should be faster.
            3. OSS-REQ-0055: may be better to rewrite this as a rate as well, although acceptable as is
            4. Remove OSS-REQ-0054.
            Show
            ktl Kian-Tat Lim added a comment - For the OSS, my proposal is: OSS-REQ-0004: change "reprocessing" to "processing" (leftover from AP move). OSS-REQ-0052: summitBufferTransferTime should be rewritten as a transfer rate (2 days per day).  Note that DMS-REQ-0165 catches up processing at one day per day, but transfer can and should be faster. OSS-REQ-0055: may be better to rewrite this as a rate as well, although acceptable as is Remove OSS-REQ-0054.
            Hide
            tjenness Tim Jenness added a comment -

            Marking as done since a new LCR has been proposed: https://project.lsst.org/groups/ccb/node/2238

            Show
            tjenness Tim Jenness added a comment - Marking as done since a new LCR has been proposed: https://project.lsst.org/groups/ccb/node/2238

              People

              • Assignee:
                tjenness Tim Jenness
                Reporter:
                gpdf Gregory Dubois-Felsmann
                Watchers:
                Gregory Dubois-Felsmann, Jeff Kantor, Kian-Tat Lim, Tim Jenness
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Due:
                  Created:
                  Updated:
                  Resolved:

                  Summary Panel