Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-36931

rotation data increasingly delayed over time

    XMLWordPrintable

Details

    • Story
    • Status: Done
    • Resolution: Done
    • None
    • ts_main_telescope

    Description

      The rotation telemetry topic is increasingly delayed over time.

      I added a diagnostic print statement to the MTRotator CSC in an effort to figure out if this was likely in the low-level controller or the CSC. The statement is in the callback function that is called whenever the CSC reads telemetry from the low-level controller. It prints two values:

      • telemetry_age: rcv_time - telemetry header timestamp, where rcv_time is the time at which the telemetry calllback function is called to process low-level telemetry.
      • write rotation delay: the time just after the "rotation" SAL event is sent - rcv_time.

      What I see is that relemetry_tag increases slowly and apparently fairly linearly, whereas write rotation delay does not change.

      telemetry_age=0.084; write rotation delay=0.001
      telemetry_age=0.084; write rotation delay=0.001
      telemetry_age=0.084; write rotation delay=0.001
      telemetry_age=0.085; write rotation delay=0.001
      telemetry_age=0.085; write rotation delay=0.001
      telemetry_age=0.085; write rotation delay=0.001
      telemetry_age=0.086; write rotation delay=0.001
      telemetry_age=0.086; write rotation delay=0.001
      telemetry_age=0.086; write rotation delay=0.001
      telemetry_age=0.087; write rotation delay=0.001
      telemetry_age=0.087; write rotation delay=0.001
      telemetry_age=0.087; write rotation delay=0.001
      telemetry_age=0.088; write rotation delay=0.001
      telemetry_age=0.088; write rotation delay=0.001
      telemetry_age=0.088; write rotation delay=0.001
      telemetry_age=0.089; write rotation delay=0.001
      telemetry_age=0.089; write rotation delay=0.001
      telemetry_age=0.089; write rotation delay=0.001
      telemetry_age=0.090; write rotation delay=0.001
      telemetry_age=0.090; write rotation delay=0.001
      telemetry_age=0.091; write rotation delay=0.001
      telemetry_age=0.091; write rotation delay=0.001
      telemetry_age=0.091; write rotation delay=0.001
      telemetry_age=0.092; write rotation delay=0.001
      telemetry_age=0.092; write rotation delay=0.001
      telemetry_age=0.092; write rotation delay=0.001
      telemetry_age=0.093; write rotation delay=0.001
      telemetry_age=0.093; write rotation delay=0.001
      telemetry_age=0.093; write rotation delay=0.001
      telemetry_age=0.094; write rotation delay=0.001
      telemetry_age=0.094; write rotation delay=0.001
      telemetry_age=0.094; write rotation delay=0.001
      telemetry_age=0.095; write rotation delay=0.001
      telemetry_age=0.095; write rotation delay=0.001
      telemetry_age=0.095; write rotation delay=0.001
      telemetry_age=0.096; write rotation delay=0.001
      telemetry_age=0.096; write rotation delay=0.001
      telemetry_age=0.097; write rotation delay=0.001
      telemetry_age=0.097; write rotation delay=0.001
      telemetry_age=0.097; write rotation delay=0.001
      telemetry_age=0.098; write rotation delay=0.001
      telemetry_age=0.098; write rotation delay=0.001
      telemetry_age=0.098; write rotation delay=0.001
      telemetry_age=0.099; write rotation delay=0.001
      telemetry_age=0.099; write rotation delay=0.001
      telemetry_age=0.099; write rotation delay=0.001
      telemetry_age=0.100; write rotation delay=0.001
      ...
      

      The message is printed every 25 telemetry samples. After many minutes the time is up to 0.465 seconds.

      I strongly suspect the error is inside the low-level controller and I hope you will be willing to rule this out before I look for it in the CSC. I'm having trouble imagining how the CSC could do this (especially with such large delays).

      The only possible explanation I have come up with is an algorithm such as this:

      timestamp = get_current_time()
      while True:
          compute and send telemetry
          sleep telemetry_interval
          timestamp = old_timestamp + telemetry_interval
      

      The timestamp would fall further behind in each iteration by the amount of the time spent in the "compute and send telemetry" step.

      Attachments

        1. delay times after Te-Wei's fix.txt
          33 kB
        2. diff_sal_rcvStamp.png
          diff_sal_rcvStamp.png
          17 kB
        3. diff_sal_sndStamp_after_90sec.png
          diff_sal_sndStamp_after_90sec.png
          18 kB
        4. diff_sal_sndStamp.png
          diff_sal_sndStamp.png
          18 kB
        5. show_mismatched_delays_modified.py
          0.8 kB
        6. show_mismatched_delays.py
          0.7 kB
        7. timestamp_rotator_90sec.ipynb
          290 kB
        8. timestamp_rotator.ipynb
          288 kB

        Activity

          No work has yet been logged on this issue.

          People

            ttsai Te-Wei Tsai
            rowen Russell Owen
            Russell Owen
            Russell Owen, Te-Wei Tsai
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Jenkins

                No builds found.