Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-19243

Investigate Kafka for Influx EFD implementation

    XMLWordPrintable

    Details

      Description

      Investigate Kafka for Influx EFD implementation.

        Attachments

          Activity

          Hide
          aanania Andres Anania [X] (Inactive) added a comment -

          As part of this task I will work on a setup of containers for EFDWriters + Database (MariaDB and Influx), then I can introduce kafka to the same environment.

          Show
          aanania Andres Anania [X] (Inactive) added a comment - As part of this task I will work on a setup of containers for EFDWriters + Database (MariaDB and Influx), then I can introduce kafka to the same environment.
          Hide
          aanania Andres Anania [X] (Inactive) added a comment -

          I have created a dockerfile to deploy EFDWriters from the RPM:

          https://project.lsst.org/ts/lsstrepo/ts_EFDruntime-3.9.0-690.el7.centos.x86_64.rpm

          There are mutiple dependency issues: 

          1. .The LD_LIBRARY_PATH is not pointing to the right directory
          2. Missing mysql library for mariadb efd writers: (```./sacpp_ATDome_event_efdwriter: error while loading shared libraries: libmysqlclient.so.18: cannot open shared object file: No such file or directory```)
          3. Missing mysql library for kafka efd writers

          Fixes:

          1. Update in setupEFD.env  to use the right path: export LD_LIBRARY_PATH=${SAL_WORK_DIR}/lib:${LSST_SDK_INSTALL}/lib
          2. add mariadb-devel to dependencies
          3. add librdkafka to dependencies
          Show
          aanania Andres Anania [X] (Inactive) added a comment - I have created a dockerfile to deploy EFDWriters from the RPM: https://project.lsst.org/ts/lsstrepo/ts_EFDruntime-3.9.0-690.el7.centos.x86_64.rpm There are mutiple dependency issues:  .The LD_LIBRARY_PATH is not pointing to the right directory Missing mysql library for mariadb efd writers: (```./sacpp_ATDome_event_efdwriter: error while loading shared libraries: libmysqlclient.so.18: cannot open shared object file: No such file or directory```) Missing mysql library for kafka efd writers Fixes: Update in setupEFD.env  to use the right path: export LD_LIBRARY_PATH=${SAL_WORK_DIR}/lib:${LSST_SDK_INSTALL}/lib add mariadb-devel to dependencies add librdkafka to dependencies
          Hide
          aanania Andres Anania [X] (Inactive) added a comment -

          created dockerfiles on github for efd_writers, they are in: https://github.com/lsst-ts/ts_Dockerfiles/tree/tickets/DM-19243

          Also pushed efd_writers containers in: https://cloud.docker.com/u/lsstts/repository/docker/lsstts/efd_writers

          Show
          aanania Andres Anania [X] (Inactive) added a comment - created dockerfiles on github for efd_writers, they are in:  https://github.com/lsst-ts/ts_Dockerfiles/tree/tickets/DM-19243 Also pushed efd_writers containers in:  https://cloud.docker.com/u/lsstts/repository/docker/lsstts/efd_writers
          Hide
          dmills Dave Mills added a comment -

          I have updated the ts_EFD rpm on the project.lsst.org/ts/lsstrepo to require mariadb-devel and librdkafka,
          and changed the LD_LIBRARY_PATH in setupEFD.env

          Show
          dmills Dave Mills added a comment - I have updated the ts_EFD rpm on the project.lsst.org/ts/lsstrepo to require mariadb-devel and librdkafka, and changed the LD_LIBRARY_PATH in setupEFD.env
          Hide
          aanania Andres Anania [X] (Inactive) added a comment -

          reviewed all the list in the following youtube channel to learn about Kafka:

          https://www.youtube.com/watch?v=kDx8hZhvCQ0&list=PLkz1SCf5iB4enAR00Z46JwY9GGkaS2NON&index=7

          Also created docker images in lsstts/ for efd_writers and have docekr compose to create mariadb as well.

           

          Show
          aanania Andres Anania [X] (Inactive) added a comment - reviewed all the list in the following youtube channel to learn about Kafka: https://www.youtube.com/watch?v=kDx8hZhvCQ0&list=PLkz1SCf5iB4enAR00Z46JwY9GGkaS2NON&index=7 Also created docker images in lsstts/ for efd_writers and have docekr compose to create mariadb as well.  
          Hide
          aclements Andy Clements added a comment -

          Andres did a bunch of research for the Kafka Influx implementation, but we have since switched directions and are using DM implementation.  This research still will apply for stuff we do later on the down the road.

          Show
          aclements Andy Clements added a comment - Andres did a bunch of research for the Kafka Influx implementation, but we have since switched directions and are using DM implementation.  This research still will apply for stuff we do later on the down the road.

            People

            Assignee:
            aanania Andres Anania [X] (Inactive)
            Reporter:
            aanania Andres Anania [X] (Inactive)
            Reviewers:
            Andy Clements
            Watchers:
            Andres Anania [X] (Inactive), Andres Villalobos [X] (Inactive), Andy Clements, Christopher Stephens [X] (Inactive), Dave Mills, Te-Wei Tsai
            Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved:

                Jenkins

                No builds found.