Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-18513

Install the MySQL Cluster with Puppet on local machine

    XMLWordPrintable

    Details

      Description

      Install the MySQL Cluster with Puppet on local machine in Tucson.

      This is to be familiar with the baseline software setup. This task will also try to compare the MySQL Cluster and InfluxDB (NoSQL) and discuss with Andres.

      The puppet script by Andres V. is here:

      https://github.com/avillalobos/lsst_devops

        Attachments

        1. createTables.txt
          2 kB
        2. fake_telemetry_cam_pub.py
          2 kB
        3. image-2019-03-25-16-54-12-233.png
          image-2019-03-25-16-54-12-233.png
          161 kB
        4. image-2019-03-27-10-33-25-476.png
          image-2019-03-27-10-33-25-476.png
          97 kB
        5. image-2019-03-27-15-05-51-095.png
          image-2019-03-27-15-05-51-095.png
          53 kB
        6. image-2019-03-27-15-05-54-059.png
          image-2019-03-27-15-05-54-059.png
          53 kB
        7. Practical Time Series Analysis MemSQL.pdf
          7.05 MB
        8. tableFull.png
          tableFull.png
          144 kB

          Issue Links

            Activity

            Hide
            ttsai Te-Wei Tsai added a comment -

            Read the chapter 2 of Practical Time Series Analysis MemSQL.pdf to understand the consideration between SQL and NoSQL.

            Show
            ttsai Te-Wei Tsai added a comment - Read the chapter 2 of  Practical Time Series Analysis MemSQL.pdf  to understand the consideration between SQL and NoSQL.
            Hide
            ttsai Te-Wei Tsai added a comment - - edited

            I checked the number of data in ATCamera_bonnShutter, ATCamera_wreb, and ATCamera_wrebPower on ts-efd-srv-01, and the values are 953088, 953190, and 953040 individually.

            Check the memory usage by "free -m" as the following (unit is MB):

            <ts-efd-srv-01> total: 991, used: 409, available: 318.

            <ts-efd-data-01> total: 991, used: 665, available: 144.

            The memory of data is set to be 512 MB (DataMemory = 512M) on ts-efd-mgmt-01.

             

            Check the data size in ts-efd-srv-01 (/mnt/tier1), the value is 135 MB.

            Check the data size in ts-efd-data-01 (/mnt/tier1), the value is 807 MB.

             

            The disk sizes VBs on "/dev/sda1"  are 43 GB. Therefore, the disk size should be enough.

            I got the following message after the execution of ndb_mgm -e "ALL REPORT MEMORYUSAGE" under ts-efd-mgmt-01:

            Connected to Management Server at: localhost:1186
            Node 16: Data usage is 94%(14254 32K pages of total 15074)
            Node 16: Index usage is 61%(1310 32K pages of total 2130)

            The Node 16 is "ts-efd-data-01".

            The usage is 94% here because the 5% is reserved for the "restart". The default value of MinFreePct is 5, which means 5%.

            Check the data in ts-efd-data-01, there is 521 MB under "/mnt/tier1/data/ndb_16_fs/LCP". This looks like the stored data.

            Show
            ttsai Te-Wei Tsai added a comment - - edited I checked the number of data in ATCamera_bonnShutter, ATCamera_wreb, and ATCamera_wrebPower on ts-efd-srv-01, and the values are 953088, 953190, and 953040 individually. Check the memory usage by "free -m" as the following (unit is MB): <ts-efd-srv-01> total: 991, used: 409, available: 318. <ts-efd-data-01> total: 991, used: 665, available: 144. The memory of data is set to be 512 MB (DataMemory = 512M) on ts-efd-mgmt-01.   Check the data size in ts-efd-srv-01 (/mnt/tier1), the value is 135 MB. Check the data size in ts-efd-data-01 (/mnt/tier1), the value is 807 MB.   The disk sizes VBs on "/dev/sda1"  are 43 GB. Therefore, the disk size should be enough. I got the following message after the execution of ndb_mgm -e "ALL REPORT MEMORYUSAGE" under ts-efd-mgmt-01: Connected to Management Server at: localhost:1186 Node 16: Data usage is 94%(14254 32K pages of total 15074) Node 16: Index usage is 61%(1310 32K pages of total 2130) The Node 16 is "ts-efd-data-01". The usage is 94% here because the 5% is reserved for the "restart". The default value of MinFreePct is 5, which means 5%. Check the data in ts-efd-data-01, there is 521 MB under "/mnt/tier1/data/ndb_16_fs/LCP". This looks like the stored data.
            Hide
            ttsai Te-Wei Tsai added a comment - - edited

            Based on https://dev.mysql.com/doc/refman/5.7/en/mysql-cluster-ndbd-definition.html#ndbparam-ndbd-datamemory, the "PARTITION_BALANCE" is something valuable to try. The "MinFreePct" is valuable to use.

            Show
            ttsai Te-Wei Tsai added a comment - - edited Based on https://dev.mysql.com/doc/refman/5.7/en/mysql-cluster-ndbd-definition.html#ndbparam-ndbd-datamemory , the "PARTITION_BALANCE" is something valuable to try. The "MinFreePct" is valuable to use.
            Hide
            ttsai Te-Wei Tsai added a comment -

            Discussed with Andres A. and he suggested the following page to clean the data in RAM for the test purpose:

            https://confluence.lsstcorp.org/display/LTS/MySQL+Internal+configuration

             

            Show
            ttsai Te-Wei Tsai added a comment - Discussed with Andres A. and he suggested the following page to clean the data in RAM for the test purpose: https://confluence.lsstcorp.org/display/LTS/MySQL+Internal+configuration  
            Hide
            aanania Andres Anania [X] (Inactive) added a comment -

            We discussed about different technologies, Te-Wei got familiar with current puppet setup to install a MySQL Cluster. He talked to Andres V to get more familiar with the installation procedure in Puppet + Vargrant VMs. 

            Show
            aanania Andres Anania [X] (Inactive) added a comment - We discussed about different technologies, Te-Wei got familiar with current puppet setup to install a MySQL Cluster. He talked to Andres V to get more familiar with the installation procedure in Puppet + Vargrant VMs. 

              People

              Assignee:
              ttsai Te-Wei Tsai
              Reporter:
              ttsai Te-Wei Tsai
              Reviewers:
              Andres Anania [X] (Inactive)
              Watchers:
              Andres Anania [X] (Inactive), Te-Wei Tsai
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:
                Start date:
                End date:

                  Jenkins

                  No builds found.