Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-36861

The MT rotator is going in the wrong direction when tracking ramps

    XMLWordPrintable

    Details

      Description

      In the last two days every time we try a ramp tracking sequence for the MT rotator from the MTRotator commander, the rotator moves in the wrong direction, often quite far in the wrong direction. Here is an example, with a very short ramp (so I didn't have to stop it):

      MT Rotator failed ramp 2022-11-03

      ramp -25 -26 1
      run_command running command method <bound method RotatorCommander.do_ramp of <lsst.ts.mtrotator.rotator_commander.RotatorCommander object at 0x7fc1390205e0>>
      Tracking a ramp from -25.0 to -26.0 at speed 1.0; this will take 1.00 seconds
      Finished command ramp
      track angle=-25.00 velocity=-1.00
      1667506455.090: controllerState: controllerState=2, offlineSubstate=1, enabledSubstate=2, applicationStatus=5124
      1667506455.090: tracking: tracking=False, lost=True, noNewCommand=False
      1667506455.091: target: position=-25.0000, velocity=-1.0000, tai=1667506455.0936
      1667506455.090: inPosition: inPosition=False
      track angle=-25.15 velocity=-1.00
      1667506455.240: rotation: demandPosition=-25.0000, demandVelocity=-0.0000, demandAcceleration=-0.0600, actualPosition=-25.0000, actualVelocity=-0.0000, debugActualVelocityA=-0.0000, debugActualVelocityB=-0.0000, odometer=392282.8500, timestamp=1667506455.1500
      1667506455.240: tracking: tracking=True, lost=True, noNewCommand=False
      1667506455.240: inPosition: inPosition=True
      1667506455.290: rotation: demandPosition=-25.0000, demandVelocity=-0.0100, demandAcceleration=-0.1200, actualPosition=-25.0000, actualVelocity=-0.0000, debugActualVelocityA=-0.0000, debugActualVelocityB=-0.0000, odometer=392282.8500, timestamp=1667506455.2000
      1667506455.290: target: position=-25.1530, velocity=-1.0000, tai=1667506455.2466
      1667506455.340: rotation: demandPosition=-25.0000, demandVelocity=-0.0100, demandAcceleration=-0.1200, actualPosition=-25.0000, actualVelocity=-0.0200, debugActualVelocityA=-0.0200, debugActualVelocityB=-0.0100, odometer=392282.8500, timestamp=1667506455.2500
      1667506455.340: ccwFollowingError: positionError=-0.0000, velocityError=-0.0200, timestamp=1667506455.2500
      1667506455.390: rotation: demandPosition=-25.0000, demandVelocity=-0.0200, demandAcceleration=-0.1200, actualPosition=-25.0000, actualVelocity=-0.0300, debugActualVelocityA=-0.0300, debugActualVelocityB=-0.0300, odometer=392282.8600, timestamp=1667506455.3000
      1667506455.390: motors: raw=[-2453700000.0, -2451600000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
      1667506455.390: ccwFollowingError: positionError=-0.0000, velocityError=-0.0300, timestamp=1667506455.3000
      track angle=-25.35 velocity=-1.00
      1667506455.440: rotation: demandPosition=-25.0000, demandVelocity=-0.0300, demandAcceleration=-0.1200, actualPosition=-25.0000, actualVelocity=-0.0300, debugActualVelocityA=-0.0300, debugActualVelocityB=-0.0300, odometer=392282.8600, timestamp=1667506455.3500
      1667506455.490: motors: raw=[-2453600000.0, -2451500000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
      1667506455.490: target: position=-25.3514, velocity=-1.0000, tai=1667506455.4450
      1667506455.540: rotation: demandPosition=-25.0100, demandVelocity=-0.0400, demandAcceleration=-0.1200, actualPosition=-25.0100, actualVelocity=-0.0400, debugActualVelocityA=-0.0400, debugActualVelocityB=-0.0400, odometer=392282.8600, timestamp=1667506455.4500
      1667506455.540: motors: raw=[-2453600000.0, -2451400000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
      1667506455.540: ccwFollowingError: positionError=-0.0100, velocityError=-0.0400, timestamp=1667506455.4500
      1667506455.590: motors: raw=[-2453500000.0, -2451400000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
      track angle=-25.55 velocity=-1.00
      1667506455.640: rotation: demandPosition=-25.0100, demandVelocity=-0.0500, demandAcceleration=-0.1200, actualPosition=-25.0100, actualVelocity=-0.0500, debugActualVelocityA=-0.0500, debugActualVelocityB=-0.0500, odometer=392282.8600, timestamp=1667506455.5500
      1667506455.640: motors: raw=[-2453400000.0, -2451300000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
      1667506455.640: ccwFollowingError: positionError=-0.0100, velocityError=-0.0500, timestamp=1667506455.5500
      1667506455.690: rotation: demandPosition=-25.0100, demandVelocity=-0.0600, demandAcceleration=-0.1200, actualPosition=-25.0100, actualVelocity=-0.0600, debugActualVelocityA=-0.0600, debugActualVelocityB=-0.0600, odometer=392282.8700, timestamp=1667506455.6000
      1667506455.690: motors: raw=[-2453300000.0, -2451200000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
      1667506455.690: ccwFollowingError: positionError=-0.0100, velocityError=-0.0600, timestamp=1667506455.6000
      1667506455.690: target: position=-25.5512, velocity=-1.0000, tai=1667506455.6448
      1667506455.740: rotation: demandPosition=-25.0200, demandVelocity=-0.0600, demandAcceleration=-0.1200, actualPosition=-25.0200, actualVelocity=-0.0600, debugActualVelocityA=-0.0600, debugActualVelocityB=-0.0600, odometer=392282.8700, timestamp=1667506455.6500
      1667506455.740: motors: raw=[-2453200000.0, -2451100000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
      1667506455.741: ccwFollowingError: positionError=-0.0200, velocityError=-0.0600, timestamp=1667506455.6500
      1667506455.790: rotation: demandPosition=-25.0200, demandVelocity=-0.0700, demandAcceleration=-0.1200, actualPosition=-25.0200, actualVelocity=-0.0700, debugActualVelocityA=-0.0700, debugActualVelocityB=-0.0700, odometer=392282.8700, timestamp=1667506455.7000
      1667506455.790: motors: raw=[-2453100000.0, -2451000000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
      1667506455.791: ccwFollowingError: positionError=-0.0200, velocityError=-0.0700, timestamp=1667506455.7000
      track angle=-25.75 velocity=-1.00
      1667506455.840: rotation: demandPosition=-25.0200, demandVelocity=-0.0800, demandAcceleration=-0.1200, actualPosition=-25.0200, actualVelocity=-0.0800, debugActualVelocityA=-0.0800, debugActualVelocityB=-0.0800, odometer=392282.8800, timestamp=1667506455.7500
      1667506455.840: motors: raw=[-2452900000.0, -2450800000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
      1667506455.840: ccwFollowingError: positionError=-0.0200, velocityError=-0.0800, timestamp=1667506455.7500
      1667506455.890: rotation: demandPosition=-25.0300, demandVelocity=-0.0800, demandAcceleration=-0.1200, actualPosition=-25.0300, actualVelocity=-0.0800, debugActualVelocityA=-0.0800, debugActualVelocityB=-0.0800, odometer=392282.8800, timestamp=1667506455.8000
      1667506455.890: motors: raw=[-2452800000.0, -2450700000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
      1667506455.890: ccwFollowingError: positionError=-0.0300, velocityError=-0.0800, timestamp=1667506455.8000
      1667506455.890: target: position=-25.7517, velocity=-1.0000, tai=1667506455.8453
      1667506455.940: rotation: demandPosition=-25.0300, demandVelocity=-0.0900, demandAcceleration=-0.1200, actualPosition=-25.0300, actualVelocity=-0.0900, debugActualVelocityA=-0.0900, debugActualVelocityB=-0.0800, odometer=392282.8800, timestamp=1667506455.8500
      1667506455.940: motors: raw=[-2452600000.0, -2450500000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
      1667506455.940: ccwFollowingError: positionError=-0.0300, velocityError=-0.0900, timestamp=1667506455.8500
      1667506455.990: rotation: demandPosition=-25.0400, demandVelocity=-0.0900, demandAcceleration=-0.1200, actualPosition=-25.0400, actualVelocity=-0.0900, debugActualVelocityA=-0.0900, debugActualVelocityB=-0.0900, odometer=392282.8900, timestamp=1667506455.9000
      1667506455.990: motors: raw=[-2452500000.0, -2450300000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
      1667506455.990: ccwFollowingError: positionError=-0.0400, velocityError=-0.0900, timestamp=1667506455.9000
      track angle=-25.95 velocity=-1.00
      1667506456.040: rotation: demandPosition=-25.0400, demandVelocity=-0.1000, demandAcceleration=-0.1100, actualPosition=-25.0400, actualVelocity=-0.1000, debugActualVelocityA=-0.1000, debugActualVelocityB=-0.1000, odometer=392282.8900, timestamp=1667506455.9500
      1667506456.040: motors: raw=[-2452300000.0, -2450200000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
      1667506456.041: ccwFollowingError: positionError=-0.0400, velocityError=-0.1000, timestamp=1667506455.9500
      1667506456.090: rotation: demandPosition=-25.0500, demandVelocity=-0.1000, demandAcceleration=-0.0500, actualPosition=-25.0500, actualVelocity=-0.1000, debugActualVelocityA=-0.1000, debugActualVelocityB=-0.1100, odometer=392282.9000, timestamp=1667506456.0000
      1667506456.090: motors: raw=[-2452100000.0, -2450000000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
      1667506456.090: ccwFollowingError: positionError=-0.0500, velocityError=-0.1100, timestamp=1667506456.0000
      1667506456.091: target: position=-25.9516, velocity=-1.0000, tai=1667506456.0452
      1667506456.140: rotation: demandPosition=-25.0500, demandVelocity=-0.1000, demandAcceleration=0.0200, actualPosition=-25.0500, actualVelocity=-0.1000, debugActualVelocityA=-0.1100, debugActualVelocityB=-0.1000, odometer=392282.9000, timestamp=1667506456.0500
      1667506456.140: motors: raw=[-2451900000.0, -2449800000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
      1667506456.140: ccwFollowingError: positionError=-0.0500, velocityError=-0.1000, timestamp=1667506456.0500
      1667506456.190: rotation: demandPosition=-25.0600, demandVelocity=-0.1000, demandAcceleration=0.1600, actualPosition=-25.0600, actualVelocity=-0.1000, debugActualVelocityA=-0.1000, debugActualVelocityB=-0.1000, odometer=392282.9100, timestamp=1667506456.1000
      1667506456.190: motors: raw=[-2451700000.0, -2449600000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
      track angle=-26.00 velocity=0.00
      1667506456.240: rotation: demandPosition=-25.0600, demandVelocity=-0.0900, demandAcceleration=0.3000, actualPosition=-25.0600, actualVelocity=-0.0900, debugActualVelocityA=-0.0900, debugActualVelocityB=-0.1000, odometer=392282.9100, timestamp=1667506456.1500
      1667506456.240: motors: raw=[-2451500000.0, -2449400000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
      1667506456.240: ccwFollowingError: positionError=-0.0600, velocityError=-0.0900, timestamp=1667506456.1500
      1667506456.290: rotation: demandPosition=-25.0600, demandVelocity=-0.0700, demandAcceleration=0.3700, actualPosition=-25.0600, actualVelocity=-0.0700, debugActualVelocityA=-0.0700, debugActualVelocityB=-0.0700, odometer=392282.9200, timestamp=1667506456.2000
      1667506456.290: motors: raw=[-2451400000.0, -2449200000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
      1667506456.291: ccwFollowingError: positionError=-0.0600, velocityError=-0.0700, timestamp=1667506456.2000
      1667506456.290: tracking: tracking=False, lost=True, noNewCommand=False
      1667506456.291: target: position=-26.0000, velocity=0.0000, tai=1667506456.2452
      1667506456.290: inPosition: inPosition=False
      

      This shows an attempt to track a path from -25 to -26 degrees at 1 deg/second (thus a very short ramp). The rotator starts moving in the correct direction but soon turns around. The tracking commands look like this: track angle=-25.95 velocity=-1.00 (though you can also get the same information from the "target: " events, which the CSC publishes when it sends a track command or point-to-point move command to the low-level controller).

      I also attached a graph from Julen Garcia showing position vs time (of both rotator and camera cable wrap). This shows the rotator starting off in the correct direction, but then turning around and ending up in the wrong direction from the starting point. Also notice that the end of the motion is very abrupt; I doubt it is properly jerk-limited.

        Attachments

        1. rotator ramp path.png
          rotator ramp path.png
          261 kB
        2. Rotator PosVelAcc.jpg
          Rotator PosVelAcc.jpg
          193 kB
        3. failed ramp.txt
          235 kB

          Issue Links

            Activity

            Hide
            ttsai Te-Wei Tsai added a comment -

            Please help to review the PR:
            https://github.com/lsst-ts/ts_rotator_controller/pull/47

            Thanks!

            Show
            ttsai Te-Wei Tsai added a comment - Please help to review the PR: https://github.com/lsst-ts/ts_rotator_controller/pull/47 Thanks!
            Hide
            rowen Russell Owen added a comment -

            I would not like close this until we can test it. The code from Tekniker that runs on the rotator is similar to the code in the TMA, but not identical.

            Show
            rowen Russell Owen added a comment - I would not like close this until we can test it. The code from Tekniker that runs on the rotator is similar to the code in the TMA, but not identical.
            Hide
            ttsai Te-Wei Tsai added a comment -

            In this case, I would prefer:
            1. You still review that PR because I would like to put the latest Simulink update by Tekniker (which is the same as TMA base on Ismael) to the develop branch. Although I could create another ticket for this, I prefer not.
            2. I put this ticket back to "To Do" since I do not know when the rotator will be recovered for us to test.

            Please help to comment. Thanks!

            Show
            ttsai Te-Wei Tsai added a comment - In this case, I would prefer: 1. You still review that PR because I would like to put the latest Simulink update by Tekniker (which is the same as TMA base on Ismael) to the develop branch. Although I could create another ticket for this, I prefer not. 2. I put this ticket back to "To Do" since I do not know when the rotator will be recovered for us to test. Please help to comment. Thanks!
            Hide
            rowen Russell Owen added a comment -

            Maybe it makes the most sense to close this one and start a new one if we still have problems. I'll review the PR next week when I'm back at work.

            Show
            rowen Russell Owen added a comment - Maybe it makes the most sense to close this one and start a new one if we still have problems. I'll review the PR next week when I'm back at work.
            Hide
            ttsai Te-Wei Tsai added a comment -

            Thanks Russell to review and approve the PR!

            Show
            ttsai Te-Wei Tsai added a comment - Thanks Russell to review and approve the PR!

              People

              Assignee:
              ttsai Te-Wei Tsai
              Reporter:
              rowen Russell Owen
              Reviewers:
              Russell Owen
              Watchers:
              Brian Stalder, Bruno Quint, Julen Garcia, Russell Owen, Te-Wei Tsai
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Jenkins

                  No builds found.