Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-17172

Update configuration to MySQL Cluster to be able to add more topics

    XMLWordPrintable

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: ts_middleware
    • Labels:

      Description

      The default configuration doesn't give enough memory to MySQL cluster to allocate all the tables for the EFD. Need to be reviewed why and how to solve it.

        Attachments

          Issue Links

            Activity

            Hide
            aanania Andres Anania [X] (Inactive) added a comment - - edited

            Added: 
            MaxNoOfOrderedIndexes = 5000    # Needed to add more tables
            MaxNoOfUniqueHashIndexes = 5000 # Needed to add more tables

            In the management node, this seems to be working fine.

            I have this configuration running with some publishers. I will keep it running for a while to check that the EFD won't have any issues after storing data.

            Show
            aanania Andres Anania [X] (Inactive) added a comment - - edited Added:  MaxNoOfOrderedIndexes =  5000      # Needed to add more tables MaxNoOfUniqueHashIndexes =  5000   # Needed to add more tables In the management node, this seems to be working fine. I have this configuration running with some publishers. I will keep it running for a while to check that the EFD won't have any issues after storing data.
            Hide
            aanania Andres Anania [X] (Inactive) added a comment -

            Tried to create the whole EFD schema but I had issues to new constraints that the NDB engine has over innoDB (512 attributes per table at max).  We will need to find a solution to this issue, either add a new constraint or create additional tables as "arrays"

             

            Show
            aanania Andres Anania [X] (Inactive) added a comment - Tried to create the whole EFD schema but I had issues to new constraints that the NDB engine has over innoDB (512 attributes per table at max).  We will need to find a solution to this issue, either add a new constraint or create additional tables as "arrays"  
            Hide
            aanania Andres Anania [X] (Inactive) added a comment -

            Created a Jira ticket to Dave to review the last comment.:

            https://jira.lsstcorp.org/browse/DM-17204

             

             

            Show
            aanania Andres Anania [X] (Inactive) added a comment - Created a Jira ticket to Dave to review the last comment.: https://jira.lsstcorp.org/browse/DM-17204    
            Hide
            aanania Andres Anania [X] (Inactive) added a comment - - edited

            It's been working for a week now. I couldn't create all the scheme due to ndb limitation in table size but that is under review now. 

            To test you can ssh to tcs@139.229.162.113 (nice password), go to the database:
            >mysql -uefduser -p 
            and do a 

            >show tables 

            Also if you do multiple time a: 

            mysql>select count( *) from ATMCS_mountEncoders; 

            you can see that the data keeps increasing.

            Show
            aanania Andres Anania [X] (Inactive) added a comment - - edited It's been working for a week now. I couldn't create all the scheme due to ndb limitation in table size but that is under review now.  To test you can ssh to tcs@139.229.162.113 (nice password), go to the database: >mysql -uefduser -p  and do a  >show tables  Also if you do multiple time a:  mysql>select count( *) from ATMCS_mountEncoders;  you can see that the data keeps increasing.
            Hide
            aanania Andres Anania [X] (Inactive) added a comment - - edited

            You can also check Andres V grafana to see EFD performance in: 

            http://gs-grafana-node-01.po.us.lsst.org:3000/d/N-bxMeUiz/efd-monitoring?refresh=5s&orgId=1

            User and pass will be sent internally 

            Show
            aanania Andres Anania [X] (Inactive) added a comment - - edited You can also check Andres V grafana to see EFD performance in:  http://gs-grafana-node-01.po.us.lsst.org:3000/d/N-bxMeUiz/efd-monitoring?refresh=5s&orgId=1 User and pass will be sent internally 
            Hide
            aanania Andres Anania [X] (Inactive) added a comment -

            example plot: 

            Show
            aanania Andres Anania [X] (Inactive) added a comment - example plot: 
            Hide
            dmills Dave Mills added a comment -

            Increased capacity configured and tested

            Show
            dmills Dave Mills added a comment - Increased capacity configured and tested

              People

              Assignee:
              aanania Andres Anania [X] (Inactive)
              Reporter:
              aanania Andres Anania [X] (Inactive)
              Reviewers:
              Dave Mills
              Watchers:
              Andres Anania [X] (Inactive), Andres Villalobos [X] (Inactive), Dave Mills
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:
                Start date:
                End date:

                  Jenkins

                  No builds found.