Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-27905

Implement the ability to handle subsets of topics in the SAL/kafka producer

    XMLWordPrintable

    Details

    • Story Points:
      5
    • Epic Link:
    • Sprint:
      TSSW Sprint - Feb 1 - Feb 15, TSSW Sprint - Feb 15 - Mar 1, TSSW Sprint - Mar 1 - Mar 15, TSSW Sprint - Mar 15 - Mar 29, TSSW Sprint - Mar 29 - Apr 12
    • Team:
      Telescope and Site
    • Urgent?:
      No

      Description

      Finish Tiago Ribeiro's work in allowing a SAL/kafka producer to handle a subset of topics. This is needed for a few very chatty SAL components.

        Attachments

          Activity

          Ascending order - Click to sort in descending order
          No builds found.
          rowen Russell Owen created issue -
          rowen Russell Owen made changes -
          Field Original Value New Value
          Epic Link DM-27708 [ 442112 ]
          rowen Russell Owen made changes -
          Sprint TSSW Sprint - Feb 1 - Feb 15 [ 1076 ]
          rowen Russell Owen made changes -
          Status To Do [ 10001 ] In Progress [ 3 ]
          Hide
          Permalink
          rowen Russell Owen added a comment - - edited

          Changes:

          • Added TopicNames class to manage a subset of topic names.
          • Added TopicNamesSet classes to manage a complete collection of subsets of topic names (TopicNames instances) for a given SAL component.
          • Updated ComponentProducer to accept a TopicNames instance
          • Refactored ComponentPoducerSet such that the constructor does very little. Call one instance method to run a normal producer and a different instance method to run a single component using subprocesses for subsets of topics.
          • Improve the command-line parser in several ways. It no longer requires kafka arguments when using --validate or --show-schema, but catches some other errors earlier.
          • Added tests/test_component_producer_set.py which tests some aspects of ComponentProducerSet, though alas not actually producing data (I was not able to figure out how to manage that).
          • Improved and modernized the documentation.

          Pull request (from tickets/DM-26581 rebased to develop): https://github.com/lsst-ts/ts_salkafka/pull/19

          Show
          rowen Russell Owen added a comment - - edited Changes: Added TopicNames class to manage a subset of topic names. Added TopicNamesSet classes to manage a complete collection of subsets of topic names (TopicNames instances) for a given SAL component. Updated ComponentProducer to accept a TopicNames instance Refactored ComponentPoducerSet such that the constructor does very little. Call one instance method to run a normal producer and a different instance method to run a single component using subprocesses for subsets of topics. Improve the command-line parser in several ways. It no longer requires kafka arguments when using --validate or --show-schema, but catches some other errors earlier. Added tests/test_component_producer_set.py which tests some aspects of ComponentProducerSet, though alas not actually producing data (I was not able to figure out how to manage that). Improved and modernized the documentation. Pull request (from tickets/ DM-26581 rebased to develop): https://github.com/lsst-ts/ts_salkafka/pull/19
          wvreeven Wouter van Reeven made changes -
          Sprint TSSW Sprint - Feb 1 - Feb 15 [ 1076 ] TSSW Sprint - Feb 1 - Feb 15, TSSW Sprint - Feb 15 - Mar 1 [ 1076, 1078 ]
          rowen Russell Owen made changes -
          Reviewers Tiago Ribeiro [ tribeiro ]
          Status In Progress [ 3 ] In Review [ 10004 ]
          rowen Russell Owen made changes -
          Story Points 3 5
          wvreeven Wouter van Reeven made changes -
          Sprint TSSW Sprint - Feb 1 - Feb 15, TSSW Sprint - Feb 15 - Mar 1 [ 1076, 1078 ] TSSW Sprint - Feb 1 - Feb 15, TSSW Sprint - Feb 15 - Mar 1, TSSW Sprint - Mar 1 - Mar 15 [ 1076, 1078, 1081 ]
          wvreeven Wouter van Reeven made changes -
          Sprint TSSW Sprint - Feb 1 - Feb 15, TSSW Sprint - Feb 15 - Mar 1, TSSW Sprint - Mar 1 - Mar 15 [ 1076, 1078, 1081 ] TSSW Sprint - Feb 1 - Feb 15, TSSW Sprint - Feb 15 - Mar 1, TSSW Sprint - Mar 1 - Mar 15, TSSW Sprint - Mar 15 - Mar 29 [ 1076, 1078, 1081, 1083 ]
          wvreeven Wouter van Reeven made changes -
          Sprint TSSW Sprint - Feb 1 - Feb 15, TSSW Sprint - Feb 15 - Mar 1, TSSW Sprint - Mar 1 - Mar 15, TSSW Sprint - Mar 15 - Mar 29 [ 1076, 1078, 1081, 1083 ] TSSW Sprint - Feb 1 - Feb 15, TSSW Sprint - Feb 15 - Mar 1, TSSW Sprint - Mar 1 - Mar 15, TSSW Sprint - Mar 15 - Mar 29, TSSW Sprint - Mar 29 - Apr 12 [ 1076, 1078, 1081, 1083, 1086 ]
          Hide
          Permalink
          tribeiro Tiago Ribeiro added a comment -

          Reviewed code in github.

          Show
          tribeiro Tiago Ribeiro added a comment - Reviewed code in github.
          tribeiro Tiago Ribeiro made changes -
          Status In Review [ 10004 ] Reviewed [ 10101 ]
          rowen Russell Owen made changes -
          Resolution Done [ 10000 ]
          Status Reviewed [ 10101 ] Done [ 10002 ]
          Hide
          Permalink
          rowen Russell Owen added a comment -

          Released as v1.6.0

          Show
          rowen Russell Owen added a comment - Released as v1.6.0

            People

            Assignee:
            rowen Russell Owen
            Reporter:
            rowen Russell Owen
            Reviewers:
            Tiago Ribeiro
            Watchers:
            Russell Owen, Tiago Ribeiro
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved:

                Jenkins

                No builds found.