Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-10024

Mini-Broker development in F18

    Details

    • Templates:
    • Epic Name:
      f18-ap-broker
    • Story Points:
      80
    • WBS:
      02C.03.03
    • Team:
      Alert Production
    • Cycle:
      Fall 2018

      Description

      This includes all non-leadership work required from the AP to complete the LDM-503-05 milestone. In addition, it covers effort to design and deploy a mini broker system which is able to:

      • Service 100 filters connected to a realistically sized LSST alert stream (ie, 1e6 alerts/night);
      • Provides resource constraints for filters (ie, demonstrate that no one filter can use all the bandwidth, all the CPU, prevent the other filters executing on time, etc);
      • Input event stream can be based on simulated events, replayed continuously at a realistic cadence.
      • Output to (at least) a filtered Kafka stream, (bonus points) to database, VOEvent stream, etc.
      • Consult with Eric Bellm to come up with some realistic, real-world filter designs to execute in this framework.
      • As part of this work, come up with a design for a system to add/remove filters to the running system. How will the filters be provided? As snippets of Python code? By somebody registering a new Docker image? Something else?

        Attachments

          Container Issues

          Stories in Epic (Custom Issue Matrix)

          Key Summary Story Points Assignee Status
           
          DM-15369

          Literature survey of alert DB technologies

          8 Maria Patterson [X] (Inactive) Done
           
          DM-15368

          Scale up number of filters per node

          4 Maria Patterson [X] (Inactive) Done
           
          DM-14838

          Deploy mini-broker prototype with Kubernetes

          10 Maria Patterson [X] (Inactive) Done
           
          DM-15053

          Prototype one-filter-per-container broker

          14 Maria Patterson [X] (Inactive) Done
           
          DM-14755

          Understand scope of restricted Python environments

          8 Maria Patterson [X] (Inactive) Done
           
          DM-14757

          Give a brown bag on the state of the broker

          8 Maria Patterson [X] (Inactive) Done
           
          DM-14812

          Make alert printer print every Nth alert

          0.5 Maria Patterson [X] (Inactive) Done
           
          DM-13913

          Stand up end-to-end alert system

          6 Maria Patterson [X] (Inactive) Done
           
          DM-13914

          Document alert distribution prototype

          6 Maria Patterson [X] (Inactive) Done
           
          DM-15680

          Provide summary of Alert Distribution design

          6 Maria Patterson [X] (Inactive) Done
           
          DM-15678

          Outline DMTN for Alert Distribution Design Doc

          2 Eric Bellm Done
           
          DM-15610

          Provide a full description of the current state of the mini-broker art

          4 Maria Patterson [X] (Inactive) Done
           
          DM-15611

          Give a brown-bag demonstration on how to operate a mini-broker

          2 Maria Patterson [X] (Inactive) Done
           
          DM-15854

          Add Mirrormaker to alert_stream repository

          2 Maria Patterson [X] (Inactive) Done
           
          DM-14756

          Investigating resource limiting in current mini-broker setup

          4 Maria Patterson [X] (Inactive) Won't Fix
           
          DM-14446

          Create mini-broker design documentation

          10 John Swinbank Invalid

            Activity

              People

              • Assignee:
                mtpatter Maria Patterson [X] (Inactive)
                Reporter:
                krughoff Simon Krughoff
                Watchers:
                Simon Krughoff
              • Votes:
                0 Vote for this issue
                Watchers:
                1 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Summary Panel