Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-10024

Mini-Broker development in F18

    Details

    • Templates:
    • Epic Name:
      f18-ap-broker
    • Story Points:
      80
    • WBS:
      02C.03.03
    • Team:
      Alert Production
    • Cycle:
      Fall 2018

      Description

      This includes all non-leadership work required from the AP to complete the LDM-503-05 milestone. In addition, it covers effort to design and deploy a mini broker system which is able to:

      • Service 100 filters connected to a realistically sized LSST alert stream (ie, 1e6 alerts/night);
      • Provides resource constraints for filters (ie, demonstrate that no one filter can use all the bandwidth, all the CPU, prevent the other filters executing on time, etc);
      • Input event stream can be based on simulated events, replayed continuously at a realistic cadence.
      • Output to (at least) a filtered Kafka stream, (bonus points) to database, VOEvent stream, etc.
      • Consult with Eric Bellm to come up with some realistic, real-world filter designs to execute in this framework.
      • As part of this work, come up with a design for a system to add/remove filters to the running system. How will the filters be provided? As snippets of Python code? By somebody registering a new Docker image? Something else?

        Attachments

        Stories in Epic (Custom Issue Matrix)

        Key Summary Story Points Assignee Status
         
        DM-15369

        Literature survey of alert DB technologies

        8 Maria Patterson [X] (Inactive) Done
         
        DM-15368

        Scale up number of filters per node

        4 Maria Patterson [X] (Inactive) Done
         
        DM-15680

        Provide summary of Alert Distribution design

        6 Maria Patterson [X] (Inactive) Done
         
        DM-15678

        Outline DMTN for Alert Distribution Design Doc

        2 Eric Bellm Done
         
        DM-15611

        Give a brown-bag demonstration on how to operate a mini-broker

        2 Maria Patterson [X] (Inactive) Done
         
        DM-15610

        Provide a full description of the current state of the mini-broker art

        4 Maria Patterson [X] (Inactive) Done
         
        DM-15053

        Prototype one-filter-per-container broker

        14 Maria Patterson [X] (Inactive) Done
         
        DM-14757

        Give a brown bag on the state of the broker

        8 Maria Patterson [X] (Inactive) Done
         
        DM-14755

        Understand scope of restricted Python environments

        8 Maria Patterson [X] (Inactive) Done
         
        DM-14838

        Deploy mini-broker prototype with Kubernetes

        10 Maria Patterson [X] (Inactive) Done
         
        DM-14812

        Make alert printer print every Nth alert

        0.5 Maria Patterson [X] (Inactive) Done
         
        DM-13914

        Document alert distribution prototype

        6 Maria Patterson [X] (Inactive) Done
         
        DM-13913

        Stand up end-to-end alert system

        6 Maria Patterson [X] (Inactive) Done
         
        DM-15854

        Add Mirrormaker to alert_stream repository

        2 Maria Patterson [X] (Inactive) Done
         
        DM-14756

        Investigating resource limiting in current mini-broker setup

        4 Maria Patterson [X] (Inactive) Won't Fix
         
        DM-14446

        Create mini-broker design documentation

        10 John Swinbank Invalid

          Activity

            People

            • Assignee:
              mtpatter Maria Patterson [X] (Inactive)
              Reporter:
              krughoff Simon Krughoff
              Watchers:
              Simon Krughoff
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Summary Panel