Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-15512

Provide estimated outbound bandwidth that may be allocated from the LDF for alert streams

    Details

    • Type: Story
    • Status: To Do
    • Resolution: Unresolved
    • Fix Version/s: None
    • Component/s: Alert Production, LDF
    • Labels:
      None
    • Team:
      Data Facility

      Description

      The Alert Production pipelines will produce a real-time stream of transient alerts. The full stream will be sent to several community brokers. It will also be processed by the LSST mini-broker in the data facility, and the filtered outputs sent to individual science users.

      Both of these services rely on the availability of network bandwidth from NCSA to the public cloud. While the size of the alert stream produced by AP may be estimated to within perhaps 25%, there is currently no clear budget for the bandwidth that the Data Facility will reserve for alert stream services. This omission is now hindering both the selection of community brokers as well as capacity and operations planning for the mini-broker.

      Mario Juric reports that an original budget was developed by Mike Freemon of NCSA, perhaps based on the assumptions 2007-era Data Access Whitepaper (Document-5373). LSE-78 describes all user access to DAC services as sharing one 10 GbE link; it is unclear if this is inclusive of the full or filtered alert streams.

      This ticket is to provide an updated estimate of the outbound network bandwidth which may be reserved for near-real-time alert transport. I (Eric Bellm) suggest that the budget be shared between full streams to community brokers and the mini-broker, which will allow for future trades between the number of community brokers and the number of supported users of the LSST mini-broker.

        Attachments

          Issue Links

            Activity

            Hide
            ebellm Eric Bellm added a comment -

            While this ticket does not actually require information about the size of the alert stream, some numbers will provide useful context.

            I do not believe that there are any formal requirements on the number of full streams to community brokers that must be supported. Based on conversations with science users and broker developers I believe that ~4 community brokers would be viewed as the minimum acceptable, but that the scientific value and utilization would be enhanced by a larger number (>= 10), as more specialized services could be developed.

            https://dmtn-028.lsst.io/ provides a lower bound on the alert packet size, of 136 kB for alerts with image cutouts but no lightcurve history. The SRD requires 10^4 alerts/34 second visit (with 10^5 as a stretch goal), implying that the transmission of the full alert stream requires an average bandwidth of > 320 Mbps. However, the alert transmission likely must be bursty to meet the 60 second latency requirement.

            The mini-broker has more specific requirements: it must support at least 100 simultaneously-connected users receiving the equivalent of 20 full-sized alerts/visit, although there is no latency requirement. This implies a sustained bandwidth of >64 Mbps. However, science users have expressed strong negative reactions to the 100-user requirement. I believe a factor of 10 increase (1000 filters) is a useful goal to aim for for this service.

            Show
            ebellm Eric Bellm added a comment - While this ticket does not actually require information about the size of the alert stream, some numbers will provide useful context. I do not believe that there are any formal requirements on the number of full streams to community brokers that must be supported. Based on conversations with science users and broker developers I believe that ~4 community brokers would be viewed as the minimum acceptable, but that the scientific value and utilization would be enhanced by a larger number (>= 10), as more specialized services could be developed. https://dmtn-028.lsst.io/ provides a lower bound on the alert packet size, of 136 kB for alerts with image cutouts but no lightcurve history. The SRD requires 10^4 alerts/34 second visit (with 10^5 as a stretch goal), implying that the transmission of the full alert stream requires an average bandwidth of > 320 Mbps. However, the alert transmission likely must be bursty to meet the 60 second latency requirement. The mini-broker has more specific requirements: it must support at least 100 simultaneously-connected users receiving the equivalent of 20 full-sized alerts/visit, although there is no latency requirement. This implies a sustained bandwidth of >64 Mbps. However, science users have expressed strong negative reactions to the 100-user requirement. I believe a factor of 10 increase (1000 filters) is a useful goal to aim for for this service.

              People

              • Assignee:
                mbutler Michelle Butler
                Reporter:
                ebellm Eric Bellm
                Watchers:
                Eric Bellm, John Swinbank, Maria Patterson [X] (Inactive)
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:

                  Summary Panel