Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-18453

Deployment environtment for MySQL Cluster + CSC

    XMLWordPrintable

    Details

      Description

      Work with Andres Villalobos [X] to deploy the MySQL Cluster + CSC to replicate some of the issues we've been experimenting:

      • High memory usage

        Attachments

          Issue Links

            Activity

            Hide
            aanania Andres Anania [X] (Inactive) added a comment -

            We configured an environment and met with Nicolas from INRIA. Andres V did a demo on how to deploy MySQL cluster using his puppet scripts and seems to be working. Need to send a python script to publish data so Nicolas can fill his EFD setup.

            Show
            aanania Andres Anania [X] (Inactive) added a comment - We configured an environment and met with Nicolas from INRIA. Andres V did a demo on how to deploy MySQL cluster using his puppet scripts and seems to be working. Need to send a python script to publish data so Nicolas can fill his EFD setup.
            Hide
            aanania Andres Anania [X] (Inactive) added a comment -

            The EFD infrastracture was replicated by the vendor and they are testing with the current setup

            Show
            aanania Andres Anania [X] (Inactive) added a comment - The EFD infrastracture was replicated by the vendor and they are testing with the current setup
            Hide
            aanania Andres Anania [X] (Inactive) added a comment -

             

            Response from Nicolas from INRIA:

            Nicolas Singh [6:01 PM]

            I have to go out now @Andres Anania. I’ll ping you tomorrow once I get it ready using the latest versionNicolas Singh [10:52 PM]
            just to confirm tomorrow… I did the setup of the local stack again using master branch as of today (`7236d1c`). After spinning the `puppet-master` node and the MySQL cluster nodes (`ts-efd-mgmt`, `ts-efd-data-01` and `ts-efd-srv-01`), I created the tables involved with the python script using the following scripts (taken from `/opt/ts_sal/build/sql`) into tier1 `EFD` database:
            ```DROP TABLE IF EXISTS ATCamera_bonnShutter;
            CREATE TABLE ATCamera_bonnShutter (
              date_time DATETIME(6),
              private_revCode char(8),
              private_sndStamp double precision,
              private_rcvStamp double precision,
              private_seqNum int,
              private_origin int,
              private_host int,
              shutter5V float,
              shutter36V float,
              PRIMARY KEY (date_time)
            );DROP TABLE IF EXISTS ATCamera_wreb;
            CREATE TABLE ATCamera_wreb (
              date_time DATETIME(6),
              private_revCode char(8),
              private_sndStamp double precision,
              private_rcvStamp double precision,
              private_seqNum int,
              private_origin int,
              private_host int,
              ckPSH_V float,
              ckPOV float,
              ogoV float,
              temp1 float,
              temp2 float,
              temp3 float,
              temp4 float,
              temp5 float,
              temp6 float,
              atemp0U float,
              atemp0L float,
              ccdTemp0 float,
              rtdTemp float,
              digPS_V float,
              digPS_I float,
              anaPS_V float,
              anaPS_I float,
              clkHPS_V float,
              clkHPS_I float,
              odPS_V float,
              odPS_I float,
              htrPS_V float,
              htrPS_I float,
              power float,
              sckU_V float,
              sckL_V float,
              rgU_V float,
              rgL_V float,
              cks0V float,
              rg0V float,
              od0V float,
              rd0V float,
              gd0V float,
              od0I float,
              PRIMARY KEY (date_time)
            );DROP TABLE IF EXISTS ATCamera_wrebPower;
            CREATE TABLE ATCamera_wrebPower (
              date_time DATETIME(6),
              private_revCode char(8),
              private_sndStamp double precision,
              private_rcvStamp double precision,
              private_seqNum int,
              private_origin int,
              private_host int,
              digital_V float,
              digital_I float,
              analog_V float,
              analog_I float,
              clkHigh_V float,
              clkHigh_I float,
              clkLow_V float,
              clkLow_I float,
              od_V float,
              od_I float,
              dphi_V float,
              dphi_I float,
              heater_V float,
              heater_I float,
              hvbias_V float,
              hvbias_I float,
              PRIMARY KEY (date_time)
            );```
            (edited)
            and then I ran the python script you sent me, at `/home/salmgr` folder
            ```[salmgr@ts-efd-srv-01 ~]$ python3.6 fake_telemetry_cam_pub.py
            === [putSample] ATCamera::bonnShutter_cc3d4d32 writing a message containing :
                revCode  : cc3d4d32
            === [putSample] ATCamera::wreb_4806d7e4 writing a message containing :
                revCode  : 4806d7e4
            === [putSample] ATCamera::wrebPower_2e21c3a4 writing a message containing :
                revCode  : 2e21c3a4
            . . .```
            after a few hours, I faced this message in the mysqld server:
            ```[root@ts-efd-srv-01 system]# systemctl status efd_mysqld_tier1.service  -l
            ● efd_mysqld_tier1.service - EFD MySQL daemon
               Loaded: loaded (/etc/systemd/system/efd_mysqld_tier1.service; disabled; vendor preset: disabled)
               Active: active (running) since Tue 2019-03-19 21:02:19 UTC; 4h 29min ago
             Main PID: 10710 (mysqld)
               CGroup: /system.slice/efd_mysqld_tier1.service
                       └─10710 /sbin/mysqld --defaults-file=/etc/my.cnf.d/mysql-server-tier1.cnf --ndbcluster --datadir=/mnt/tier1/srv/Mar 20 01:31:22 ts-efd-srv-01.vm.dev.lsst.org mysqld[10710]: 2019-03-20T01:31:22.560102Z 7 [ERROR] /sbin/mysqld: The table 'ATCamera_bonnShutter' is full
            Mar 20 01:31:22 ts-efd-srv-01.vm.dev.lsst.org mysqld[10710]: 2019-03-20T01:31:22.565296Z 7 [ERROR] /sbin/mysqld: The table 'ATCamera_wreb' is full
            Mar 20 01:31:22 ts-efd-srv-01.vm.dev.lsst.org mysqld[10710]: 2019-03-20T01:31:22.567247Z 7 [ERROR] /sbin/mysqld: The table 'ATCamera_wrebPower' is full```
            is that an example of replication of this issue?
            

             

            Show
            aanania Andres Anania [X] (Inactive) added a comment -   Response from Nicolas from INRIA: Nicolas Singh [6:01 PM] I have to go out now @Andres Anania. I’ll ping you tomorrow once I get it ready using the latest versionNicolas Singh [ 10 : 52 PM] just to confirm tomorrow… I did the setup of the local stack again using master branch as of today (`7236d1c`). After spinning the `puppet-master` node and the MySQL cluster nodes (`ts-efd-mgmt`, `ts-efd-data- 01 ` and `ts-efd-srv- 01 `), I created the tables involved with the python script using the following scripts (taken from `/opt/ts_sal/build/sql`) into tier1 `EFD` database: ```DROP TABLE IF EXISTS ATCamera_bonnShutter; CREATE TABLE ATCamera_bonnShutter ( date_time DATETIME( 6 ), private_revCode char ( 8 ), private_sndStamp double precision, private_rcvStamp double precision, private_seqNum int , private_origin int , private_host int , shutter5V float , shutter36V float , PRIMARY KEY (date_time) );DROP TABLE IF EXISTS ATCamera_wreb; CREATE TABLE ATCamera_wreb ( date_time DATETIME( 6 ), private_revCode char ( 8 ), private_sndStamp double precision, private_rcvStamp double precision, private_seqNum int , private_origin int , private_host int , ckPSH_V float , ckPOV float , ogoV float , temp1 float , temp2 float , temp3 float , temp4 float , temp5 float , temp6 float , atemp0U float , atemp0L float , ccdTemp0 float , rtdTemp float , digPS_V float , digPS_I float , anaPS_V float , anaPS_I float , clkHPS_V float , clkHPS_I float , odPS_V float , odPS_I float , htrPS_V float , htrPS_I float , power float , sckU_V float , sckL_V float , rgU_V float , rgL_V float , cks0V float , rg0V float , od0V float , rd0V float , gd0V float , od0I float , PRIMARY KEY (date_time) );DROP TABLE IF EXISTS ATCamera_wrebPower; CREATE TABLE ATCamera_wrebPower ( date_time DATETIME( 6 ), private_revCode char ( 8 ), private_sndStamp double precision, private_rcvStamp double precision, private_seqNum int , private_origin int , private_host int , digital_V float , digital_I float , analog_V float , analog_I float , clkHigh_V float , clkHigh_I float , clkLow_V float , clkLow_I float , od_V float , od_I float , dphi_V float , dphi_I float , heater_V float , heater_I float , hvbias_V float , hvbias_I float , PRIMARY KEY (date_time) );``` (edited) and then I ran the python script you sent me, at `/home/salmgr` folder ```[salmgr @ts -efd-srv- 01 ~]$ python3. 6 fake_telemetry_cam_pub.py === [putSample] ATCamera::bonnShutter_cc3d4d32 writing a message containing : revCode : cc3d4d32 === [putSample] ATCamera::wreb_4806d7e4 writing a message containing : revCode : 4806d7e4 === [putSample] ATCamera::wrebPower_2e21c3a4 writing a message containing : revCode : 2e21c3a4 . . .``` after a few hours, I faced this message in the mysqld server: ```[root @ts -efd-srv- 01 system]# systemctl status efd_mysqld_tier1.service -l ● efd_mysqld_tier1.service - EFD MySQL daemon Loaded: loaded (/etc/systemd/system/efd_mysqld_tier1.service; disabled; vendor preset: disabled) Active: active (running) since Tue 2019 - 03 - 19 21 : 02 : 19 UTC; 4h 29min ago Main PID: 10710 (mysqld) CGroup: /system.slice/efd_mysqld_tier1.service └─ 10710 /sbin/mysqld --defaults-file=/etc/my.cnf.d/mysql-server-tier1.cnf --ndbcluster --datadir=/mnt/tier1/srv/Mar 20 01 : 31 : 22 ts-efd-srv- 01 .vm.dev.lsst.org mysqld[ 10710 ]: 2019 - 03 -20T01: 31 : 22 .560102Z 7 [ERROR] /sbin/mysqld: The table 'ATCamera_bonnShutter' is full Mar 20 01 : 31 : 22 ts-efd-srv- 01 .vm.dev.lsst.org mysqld[ 10710 ]: 2019 - 03 -20T01: 31 : 22 .565296Z 7 [ERROR] /sbin/mysqld: The table 'ATCamera_wreb' is full Mar 20 01 : 31 : 22 ts-efd-srv- 01 .vm.dev.lsst.org mysqld[ 10710 ]: 2019 - 03 -20T01: 31 : 22 .567247Z 7 [ERROR] /sbin/mysqld: The table 'ATCamera_wrebPower' is full``` is that an example of replication of this issue?  
            Hide
            ttsai Te-Wei Tsai added a comment -

            The vendor can reproduce the memory issue. Great job!

            Show
            ttsai Te-Wei Tsai added a comment - The vendor can reproduce the memory issue. Great job!
            Hide
            aanania Andres Anania [X] (Inactive) added a comment -

            Done

            Show
            aanania Andres Anania [X] (Inactive) added a comment - Done

              People

              Assignee:
              aanania Andres Anania [X] (Inactive)
              Reporter:
              aanania Andres Anania [X] (Inactive)
              Reviewers:
              Te-Wei Tsai
              Watchers:
              Andres Anania [X] (Inactive), Andres Villalobos [X] (Inactive), Andy Clements, Te-Wei Tsai
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Jenkins

                  No builds found.