Details
-
Type:
Story
-
Status: Done
-
Resolution: Done
-
Fix Version/s: None
-
Component/s: ts_aos
-
Labels:
-
Story Points:3
-
Epic Link:
-
Sprint:TSSW Sprint - Apr 15 - Apr 27
-
Team:Telescope and Site
Description
This task will evaluate the integration between influxDB and influx_writer of SAL. This task will also benchmark the performance of InfluxDB to use the SSD, NVME, or hard disk.
Compare the MariaDB quary language and InfluxDB quary language.
Evaluation steps:
- Check with the AT CSC owners with the telemetry rate according to the page: https://confluence.lsstcorp.org/display/SYSENG/Auxiliary+Telescope+Control+Computers+in+Tucson
- Clone the ts_sal repository to the /home/ttsai. Install the SAL and build the AT CSC test scripts I need. Make sure the sal version is 3.9 and opensplice version is 6.9.
- Change the test telemetry script to run infinity times (iseq in C++) with the correct telemetry frequency (delay_1s in C, the first argument is sec and the second is nano-sec).
- Run the test for SSD, NVME, and Disk.
- The path variable of "LSST_EFD_HOST" gives the IP_address of influxDB.
- I can do "./some_writer >& log.1 &" to run the writer in the background. And I can use "tail -f log.1 to see the log file".
- Once I changed the cpp file, I can go to the "stand_alone" directory and do "make -f Makefile.sacpp_blahblak_pub" to build the code in the same directory.
- I can also do the above modifications by python.
Attachments
Activity
| Field | Original Value | New Value |
|---|---|---|
| Epic Link |
|
| Description | This task will evaluate the integration between influxDB and influx_writer of SAL. This task will also benchmark the performance of InfluxDB to use the SSD, NVME, or hard disk. |
This task will evaluate the integration between influxDB and influx_writer of SAL. This task will also benchmark the performance of InfluxDB to use the SSD, NVME, or hard disk.
Compare the MariaDB quary language and InfluxDB quary language. |
| Description |
This task will evaluate the integration between influxDB and influx_writer of SAL. This task will also benchmark the performance of InfluxDB to use the SSD, NVME, or hard disk.
Compare the MariaDB quary language and InfluxDB quary language. |
This task will evaluate the integration between influxDB and influx_writer of SAL. This task will also benchmark the performance of InfluxDB to use the SSD, NVME, or hard disk.
Compare the MariaDB quary language and InfluxDB quary language. Evaluation steps: 1. Check with the AT CSC owners with the telemetry rate according to the page of |
| Description |
This task will evaluate the integration between influxDB and influx_writer of SAL. This task will also benchmark the performance of InfluxDB to use the SSD, NVME, or hard disk.
Compare the MariaDB quary language and InfluxDB quary language. Evaluation steps: 1. Check with the AT CSC owners with the telemetry rate according to the page of |
This task will evaluate the integration between influxDB and influx_writer of SAL. This task will also benchmark the performance of InfluxDB to use the SSD, NVME, or hard disk.
Compare the MariaDB quary language and InfluxDB quary language. Evaluation steps: # Check with the AT CSC owners with the telemetry rate according to the page: [https://confluence.lsstcorp.org/display/SYSENG/Auxiliary+Telescope+Control+Computers+in+Tucson] # |
| Description |
This task will evaluate the integration between influxDB and influx_writer of SAL. This task will also benchmark the performance of InfluxDB to use the SSD, NVME, or hard disk.
Compare the MariaDB quary language and InfluxDB quary language. Evaluation steps: # Check with the AT CSC owners with the telemetry rate according to the page: [https://confluence.lsstcorp.org/display/SYSENG/Auxiliary+Telescope+Control+Computers+in+Tucson] # |
This task will evaluate the integration between influxDB and influx_writer of SAL. This task will also benchmark the performance of InfluxDB to use the SSD, NVME, or hard disk.
Compare the MariaDB quary language and InfluxDB quary language. Evaluation steps: # Check with the AT CSC owners with the telemetry rate according to the page: [https://confluence.lsstcorp.org/display/SYSENG/Auxiliary+Telescope+Control+Computers+in+Tucson] # Clone the ts_sal repository to the /home/ttsai. Install the SAL and build the AT CSC test scripts I need. Make sure the sal version is 3.9 and openslice version is 6.9. # Change the test telemetry script to run infinity times (iseq in C++) with the correct telemetry frequency (delay_1s in C++, the first argument is sec and the second is nano-sec). # Run the test for SSD, NVME, and Disk. # The path variable of "LSST_EFD_HOST" gives the IP_address of influxDB. # I can do "./some_writer >& log.1 &" to run the writer in the background. And I can use "tail -f log.1 to see the log file". # Once I changed the cpp file, I can go to the "stand_alone" directory and do "make -f Makefile.sacpp_blahblak_pub" to build the code in the same directory. # I can also do the above modifications by python. |
| Status | To Do [ 10001 ] | In Progress [ 3 ] |
| Attachment | AT_owner [ 37637 ] |
| Attachment | AT_owner_update [ 37643 ] |
Got all AT CSC telemetry in AT_owner_update
.
| Attachment | AT_owner_update [ 37643 ] |
| Attachment | AT_owner_update [ 37658 ] |
| Attachment | AT_owner_update [ 37658 ] |
| Attachment | AT_owner_update [ 37659 ] |
| Attachment | AT_owner_update [ 37659 ] |
| Attachment | AT_owner_update [ 37660 ] |
Wrote the shell script to generate the needed AT CSC test scripts as the following:
https://github.com/lsst-ts/ts_efdEval/blob/develop/bin/buildCscTestScripts.sh
I did not see the "generated" influx DB writer that it should be. Andres helped to reproduce this error. Feedback to Dave by email already.
| Description |
This task will evaluate the integration between influxDB and influx_writer of SAL. This task will also benchmark the performance of InfluxDB to use the SSD, NVME, or hard disk.
Compare the MariaDB quary language and InfluxDB quary language. Evaluation steps: # Check with the AT CSC owners with the telemetry rate according to the page: [https://confluence.lsstcorp.org/display/SYSENG/Auxiliary+Telescope+Control+Computers+in+Tucson] # Clone the ts_sal repository to the /home/ttsai. Install the SAL and build the AT CSC test scripts I need. Make sure the sal version is 3.9 and openslice version is 6.9. # Change the test telemetry script to run infinity times (iseq in C++) with the correct telemetry frequency (delay_1s in C++, the first argument is sec and the second is nano-sec). # Run the test for SSD, NVME, and Disk. # The path variable of "LSST_EFD_HOST" gives the IP_address of influxDB. # I can do "./some_writer >& log.1 &" to run the writer in the background. And I can use "tail -f log.1 to see the log file". # Once I changed the cpp file, I can go to the "stand_alone" directory and do "make -f Makefile.sacpp_blahblak_pub" to build the code in the same directory. # I can also do the above modifications by python. |
This task will evaluate the integration between influxDB and influx_writer of SAL. This task will also benchmark the performance of InfluxDB to use the SSD, NVME, or hard disk.
Compare the MariaDB quary language and InfluxDB quary language. Evaluation steps: # Check with the AT CSC owners with the telemetry rate according to the page: [https://confluence.lsstcorp.org/display/SYSENG/Auxiliary+Telescope+Control+Computers+in+Tucson] # Clone the ts_sal repository to the /home/ttsai. Install the SAL and build the AT CSC test scripts I need. Make sure the sal version is 3.9 and opensplice version is 6.9. # Change the test telemetry script to run infinity times (iseq in C+) ++ with the correct telemetry frequency (delay_1s in C+, the first argument is sec and the second is nano-sec). # Run the test for SSD, NVME, and Disk. # The path variable of "LSST_EFD_HOST" gives the IP_address of influxDB. # I can do "./some_writer >& log.1 &" to run the writer in the background. And I can use "tail -f log.1 to see the log file". # Once I changed the cpp file, I can go to the "stand_alone" directory and do "make -f Makefile.sacpp_blahblak_pub" to build the code in the same directory. # I can also do the above modifications by python. |
| Description |
This task will evaluate the integration between influxDB and influx_writer of SAL. This task will also benchmark the performance of InfluxDB to use the SSD, NVME, or hard disk.
Compare the MariaDB quary language and InfluxDB quary language. Evaluation steps: # Check with the AT CSC owners with the telemetry rate according to the page: [https://confluence.lsstcorp.org/display/SYSENG/Auxiliary+Telescope+Control+Computers+in+Tucson] # Clone the ts_sal repository to the /home/ttsai. Install the SAL and build the AT CSC test scripts I need. Make sure the sal version is 3.9 and opensplice version is 6.9. # Change the test telemetry script to run infinity times (iseq in C+) ++ with the correct telemetry frequency (delay_1s in C+, the first argument is sec and the second is nano-sec). # Run the test for SSD, NVME, and Disk. # The path variable of "LSST_EFD_HOST" gives the IP_address of influxDB. # I can do "./some_writer >& log.1 &" to run the writer in the background. And I can use "tail -f log.1 to see the log file". # Once I changed the cpp file, I can go to the "stand_alone" directory and do "make -f Makefile.sacpp_blahblak_pub" to build the code in the same directory. # I can also do the above modifications by python. |
This task will evaluate the integration between influxDB and influx_writer of SAL. This task will also benchmark the performance of InfluxDB to use the SSD, NVME, or hard disk.
Compare the MariaDB quary language and InfluxDB quary language. Evaluation steps: # Check with the AT CSC owners with the telemetry rate according to the page: [https://confluence.lsstcorp.org/display/SYSENG/Auxiliary+Telescope+Control+Computers+in+Tucson] # Clone the ts_sal repository to the /home/ttsai. Install the SAL and build the AT CSC test scripts I need. Make sure the sal version is 3.9 and opensplice version is 6.9. # Change the test telemetry script to run infinity times (iseq in C++) with the correct telemetry frequency (delay_1s in C, the first argument is sec and the second is nano-sec). # Run the test for SSD, NVME, and Disk. # The path variable of "LSST_EFD_HOST" gives the IP_address of influxDB. # I can do "./some_writer >& log.1 &" to run the writer in the background. And I can use "tail -f log.1 to see the log file". # Once I changed the cpp file, I can go to the "stand_alone" directory and do "make -f Makefile.sacpp_blahblak_pub" to build the code in the same directory. # I can also do the above modifications by python. |
| Status | In Progress [ 3 ] | To Do [ 10001 ] |
| Priority | Major [ 3 ] | Blocker [ 1 ] |
Dave replied the email to generate the writer by tclsh. I based on this to write the following script:
https://github.com/lsst-ts/ts_efdEval/blob/develop/bin/generateInfluxDbWriter.sh
However, I still get the error and feedback to Dave already.
Dave replied that we need to use the following command to build the influxWriter:
#############################
cd $SAL_WORK_DIR
tclsh
set CSC whatever
source $env(SAL_DIR)/geninfluxefd-multi.tcl
geninfluxwriters $CSC
#############################
Writers will be in $SAL_WORK_DIR/$CSC/cpp/src
We will need the influxdb.hpp to do the building. Andres has an updated repo here:
https://github.com/aanania/influxdb-cpp
Dave mentioned I will need to put the file in "/usr/include".
| Attachment | errorFromTclsh.log [ 37669 ] | |
| Attachment | errorWhenBuildWriter.log [ 37670 ] |
I got the error (errorFromTclsh.log) When I build the influxDB writer.
After checking the generated files in ATCamera, I have the following:
When I do "make -f Makefile.sacpp_ATCamera_influxwriter", I have the error (errorWhenBuildWriter.log).
Feedback to Dave to these error.
| Attachment | error img.png [ 37671 ] |
Dave provided a new verion of influxdb.hpp. I still got the error when I tried to generate the influxdb writer with new file. Feedback to Dave already.
Andres helped to solve the problem by "yum install curl-devel". This package is not installed.
| Priority | Blocker [ 1 ] | Major [ 3 ] |
| Status | To Do [ 10001 ] | In Progress [ 3 ] |
I can write the data into the influx db already by the writer. I do not find the make file for the individual telemetry. Will check with Dave for this.
Finished the update of telemetry of CSC. Issued all influx writers and telemetries. Each CSC uses the different screen. It looks like the influxDB mostly gets the result from the first issued CSC (TunableLaser). The issues time is 10:30 am, 4/20.
Documented the evaluation in:
https://confluence.lsstcorp.org/display/LTS/InfluxDB+Evaluation
We noticed that there is the message loss for only one CSC with high frequency (10 Hz). After the discussion with Dave, he will provide a new influxwriter for the test. In this new writer, each http API by writer will send one topic value in one time.
We noticed that the InfluxDB ingested the similar amounts of data point between 20 Hz and 15 Hz. All of them are smaller than the amount of theoretical data point. Dave suggested to compare the default InfluxDB setting with DM's.
However, after the comparison, there is no big difference. Dave decided to let me to check the timestamp in InfluxDB first to try to figure out why and when the data loss.
Extracted the csv data out.
To get the final value, we can do something like: "influx -database 'EFD' -execute 'SELECT * FROM PointingComponent_timeAndDate GROUP BY * ORDER BY DESC LIMIT 1' -precision ms -format 'csv' > test.csv".
https://stackoverflow.com/questions/29193898/influxdb-getting-only-last-value-in-query
It looks like the influxwriter can only write the speed of ~0.1 sec.
| Reviewers | Dave Mills [ dmills ] | |
| Status | In Progress [ 3 ] | In Review [ 10004 ] |
Excellent summary of progress thus far. Next sprint we will add tasks
to investigate the rate limiting issue.
Dave Mills
added a comment - Excellent summary of progress thus far. Next sprint we will add tasks
to investigate the rate limiting issue. Dave Mills
made changes -
| Resolution | Done [ 10000 ] | |
| Status | In Review [ 10004 ] | Done [ 10002 ] |
Based on the AT_owner
to send the emails to ask for the AT CSC telemetry frequencies.