Details
-
Type:
Story
-
Status: Invalid
-
Resolution: Done
-
Fix Version/s: None
-
Component/s: Alert Production, LDF
-
Labels:None
-
Team:Data Facility
Description
The Alert Production pipelines will produce a real-time stream of transient alerts. The full stream will be sent to several community brokers. It will also be processed by the LSST mini-broker in the data facility, and the filtered outputs sent to individual science users.
Both of these services rely on the availability of network bandwidth from NCSA to the public cloud. While the size of the alert stream produced by AP may be estimated to within perhaps 25%, there is currently no clear budget for the bandwidth that the Data Facility will reserve for alert stream services. This omission is now hindering both the selection of community brokers as well as capacity and operations planning for the mini-broker.
Mario Juric reports that an original budget was developed by Mike Freemon of NCSA, perhaps based on the assumptions 2007-era Data Access Whitepaper (Document-5373). LSE-78 describes all user access to DAC services as sharing one 10 GbE link; it is unclear if this is inclusive of the full or filtered alert streams.
This ticket is to provide an updated estimate of the outbound network bandwidth which may be reserved for near-real-time alert transport. I (Eric Bellm) suggest that the budget be shared between full streams to community brokers and the mini-broker, which will allow for future trades between the number of community brokers and the number of supported users of the LSST mini-broker.
Attachments
Issue Links
- relates to
-
- Implemented
Activity
This is now SLAC and we had the chat with Richard/SLAC about bandwitdh. We allowed an extra broker.
While this ticket does not actually require information about the size of the alert stream, some numbers will provide useful context.
I do not believe that there are any formal requirements on the number of full streams to community brokers that must be supported. Based on conversations with science users and broker developers I believe that ~4 community brokers would be viewed as the minimum acceptable, but that the scientific value and utilization would be enhanced by a larger number (>= 10), as more specialized services could be developed.
https://dmtn-028.lsst.io/ provides a lower bound on the alert packet size, of 136 kB for alerts with image cutouts but no lightcurve history. The SRD requires 10^4 alerts/34 second visit (with 10^5 as a stretch goal), implying that the transmission of the full alert stream requires an average bandwidth of > 320 Mbps. However, the alert transmission likely must be bursty to meet the 60 second latency requirement.
The mini-broker has more specific requirements: it must support at least 100 simultaneously-connected users receiving the equivalent of 20 full-sized alerts/visit, although there is no latency requirement. This implies a sustained bandwidth of >64 Mbps. However, science users have expressed strong negative reactions to the 100-user requirement. I believe a factor of 10 increase (1000 filters) is a useful goal to aim for for this service.