Re-purpose this task due to some issues we are having with the EFD, these are:

-After a few days of test, the EFD, was giving a "Table full" error, even though the disk was almost empty. This seems to be a configuration issue which will require some research to solve properly

-Still having some issues with a limited number of attributes
-Not sure what else but I would like to write more tests to ensure that this will not keep happening

These are probably due that the EFD need some configuration optimization to handle what we need.

Problems found while creating tables from the xmls in the MySQL NDB Cluster:

(not considering MTM1M3 because it's already known that events have too many attributes for NDBCluster)

MTCamera_clusterEncoder

1. (for DomeLouvers_status)
   [root@ts-efdmysql -uefduser -plssttest EFD < efd.sqldefef
   mysql: [Warning] Using a password on the command line interface can be insecure.
   ERROR 1296 (HY000) at line 9206: Got error 4318 'Invalid attribute name or number' from NDBCLUSTER
2. (for Hexapod_logevent_settingsApplied) and probably due to a maximum size limit per table
   ```sql/Hexapod_logevent_settingsApplied.sqldef
   mysql: [Warning] Using a password on the command line interface can be insecure.
   ERROR 1296 (HY000) at line 2: Got error 1229 'Too long frm data supplied' from NDBCLUSTER``
3. (for MTCamera_clusterEncoder)
   [root@ts-efd-srv-01 build]# mysql -uefduser -plssttest EFD < efd.sqldef
   mysql: [Warning] Using a password on the command line interface can be insecure.
   ERROR 1296 (HY000) at line 14098: Got error 4318 'Invalid attribute name or number' from NDBCLUSTER
4. (for MTM2_rawTelemetry)
   mysql: [Warning] Using a password on the command line interface can be insecure.
   ERROR 1296 (HY000) at line 17550: Got error 4318 'Invalid attribute name or number' from NDBCLUSTER
5. (for MTVMS_M1M3)
   mysql: [Warning] Using a password on the command line interface can be insecure.
   ERROR 1296 (HY000) at line 21329: Got error 4318 'Invalid attribute name or number' from NDBCLUSTER
6. (for ScriptQueue_command_stopScripts and ScriptQueue_logevent_queue)
   mysql: [Warning] Using a password on the command line interface can be insecure.
   ERROR 1296 (HY000) at line 25636: Got error 4318 'Invalid attribute name or number' from NDBCLUSTER

Aside from those topics, there other topics were valid

Configure tablespaces in a virtualized environment in the .200 server 

Also documented how to create this configuration in: 

https://confluence.lsstcorp.org/display/LTS/MySQL+Internal+configuration

I'll leave the database filling with data during the weekend and check on Monday if all went good or not....

NDBCluster stores a lot of data in RAM:

https://www.fromdual.com/mysql-cluster-memory-sizing

In the link there is a section that says:

PRIMARY KEY (PK)

Every MySQL cluster table must have a primary key (PK). If you do NOT create onw, MySQL Cluster creates one for you with a size of 8 bytes. Every PK causes a hash index (HI) which has a size of 20 bytes. HI are stored in index memory while all other information are stored in data memory.
A PK also creates an ordered index (OI) unless you create it with USING HASH

Does this means that every value of the table has 20Bytes just for the index?, if so and assuming that the the PK is a datatype DATETIME (+8Bytes), each row has 28Bytes just for the index that would go directly to RAM.

Whit this said + an example I have for ATDome_command_moveAzimuth, where I have 5.136.720 rows

(Not considering data that are not indexes because they can go to Disk)

Data In RAM = (20 + 8)*5136720/1024/1024 [MB]

Data in RAM = 137MB (with commandLog is twice this value)

Then if we consider the other tables + other TIERS it seems to me that is only possible to handle a limited amount of data (if we are careful maybe just for TIER1) before the RAM goes full.

Using a concrete example (M1M3) there are 11 topics as telemetry that are published at 50Hz (there are also events at 50Hz but I’m only considering telemetry for the exercise), then:

data/day for each table = 50*60*60*24 = 4.320.000

Then for 11 topics, 11*4.320.000 = 47.520.000 or 1,27GB of RAM/day.

I may look into other engines  

Complete

Done