The rotation telemetry topic is increasingly delayed over time.
I added a diagnostic print statement to the MTRotator CSC in an effort to figure out if this was likely in the low-level controller or the CSC. The statement is in the callback function that is called whenever the CSC reads telemetry from the low-level controller. It prints two values:
- telemetry_age: rcv_time - telemetry header timestamp, where rcv_time is the time at which the telemetry calllback function is called to process low-level telemetry.
- write rotation delay: the time just after the "rotation" SAL event is sent - rcv_time.
What I see is that relemetry_tag increases slowly and apparently fairly linearly, whereas write rotation delay does not change.
The message is printed every 25 telemetry samples. After many minutes the time is up to 0.465 seconds.
I strongly suspect the error is inside the low-level controller and I hope you will be willing to rule this out before I look for it in the CSC. I'm having trouble imagining how the CSC could do this (especially with such large delays).
The only possible explanation I have come up with is an algorithm such as this:
The timestamp would fall further behind in each iteration by the amount of the time spent in the "compute and send telemetry" step.