Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-36861

The MT rotator is going in the wrong direction when tracking ramps

    XMLWordPrintable

    Details

      Description

      In the last two days every time we try a ramp tracking sequence for the MT rotator from the MTRotator commander, the rotator moves in the wrong direction, often quite far in the wrong direction. Here is an example, with a very short ramp (so I didn't have to stop it):

      MT Rotator failed ramp 2022-11-03

      ramp -25 -26 1
      		 
      run_command running command method <bound method RotatorCommander.do_ramp of <lsst.ts.mtrotator.rotator_commander.RotatorCommander object at 0x7fc1390205e0>>
      		 
      Tracking a ramp from -25.0 to -26.0 at speed 1.0; this will take 1.00 seconds
      		 
      Finished command ramp
      		 
      track angle=-25.00 velocity=-1.00
      		 
      1667506455.090: controllerState: controllerState=2, offlineSubstate=1, enabledSubstate=2, applicationStatus=5124
      		 
      1667506455.090: tracking: tracking=False, lost=True, noNewCommand=False
      		 
      1667506455.091: target: position=-25.0000, velocity=-1.0000, tai=1667506455.0936
      		 
      1667506455.090: inPosition: inPosition=False
      		 
      track angle=-25.15 velocity=-1.00
      		 
      1667506455.240: rotation: demandPosition=-25.0000, demandVelocity=-0.0000, demandAcceleration=-0.0600, actualPosition=-25.0000, actualVelocity=-0.0000, debugActualVelocityA=-0.0000, debugActualVelocityB=-0.0000, odometer=392282.8500, timestamp=1667506455.1500
      		 
      1667506455.240: tracking: tracking=True, lost=True, noNewCommand=False
      		 
      1667506455.240: inPosition: inPosition=True
      		 
      1667506455.290: rotation: demandPosition=-25.0000, demandVelocity=-0.0100, demandAcceleration=-0.1200, actualPosition=-25.0000, actualVelocity=-0.0000, debugActualVelocityA=-0.0000, debugActualVelocityB=-0.0000, odometer=392282.8500, timestamp=1667506455.2000
      		 
      1667506455.290: target: position=-25.1530, velocity=-1.0000, tai=1667506455.2466
      		 
      1667506455.340: rotation: demandPosition=-25.0000, demandVelocity=-0.0100, demandAcceleration=-0.1200, actualPosition=-25.0000, actualVelocity=-0.0200, debugActualVelocityA=-0.0200, debugActualVelocityB=-0.0100, odometer=392282.8500, timestamp=1667506455.2500
      		 
      1667506455.340: ccwFollowingError: positionError=-0.0000, velocityError=-0.0200, timestamp=1667506455.2500
      		 
      1667506455.390: rotation: demandPosition=-25.0000, demandVelocity=-0.0200, demandAcceleration=-0.1200, actualPosition=-25.0000, actualVelocity=-0.0300, debugActualVelocityA=-0.0300, debugActualVelocityB=-0.0300, odometer=392282.8600, timestamp=1667506455.3000
      		 
      1667506455.390: motors: raw=[-2453700000.0, -2451600000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
      		 
      1667506455.390: ccwFollowingError: positionError=-0.0000, velocityError=-0.0300, timestamp=1667506455.3000
      		 
      track angle=-25.35 velocity=-1.00
      		 
      1667506455.440: rotation: demandPosition=-25.0000, demandVelocity=-0.0300, demandAcceleration=-0.1200, actualPosition=-25.0000, actualVelocity=-0.0300, debugActualVelocityA=-0.0300, debugActualVelocityB=-0.0300, odometer=392282.8600, timestamp=1667506455.3500
      		 
      1667506455.490: motors: raw=[-2453600000.0, -2451500000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
      		 
      1667506455.490: target: position=-25.3514, velocity=-1.0000, tai=1667506455.4450
      		 
      1667506455.540: rotation: demandPosition=-25.0100, demandVelocity=-0.0400, demandAcceleration=-0.1200, actualPosition=-25.0100, actualVelocity=-0.0400, debugActualVelocityA=-0.0400, debugActualVelocityB=-0.0400, odometer=392282.8600, timestamp=1667506455.4500
      		 
      1667506455.540: motors: raw=[-2453600000.0, -2451400000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
      		 
      1667506455.540: ccwFollowingError: positionError=-0.0100, velocityError=-0.0400, timestamp=1667506455.4500
      		 
      1667506455.590: motors: raw=[-2453500000.0, -2451400000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
      		 
      track angle=-25.55 velocity=-1.00
      		 
      1667506455.640: rotation: demandPosition=-25.0100, demandVelocity=-0.0500, demandAcceleration=-0.1200, actualPosition=-25.0100, actualVelocity=-0.0500, debugActualVelocityA=-0.0500, debugActualVelocityB=-0.0500, odometer=392282.8600, timestamp=1667506455.5500
      		 
      1667506455.640: motors: raw=[-2453400000.0, -2451300000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
      		 
      1667506455.640: ccwFollowingError: positionError=-0.0100, velocityError=-0.0500, timestamp=1667506455.5500
      		 
      1667506455.690: rotation: demandPosition=-25.0100, demandVelocity=-0.0600, demandAcceleration=-0.1200, actualPosition=-25.0100, actualVelocity=-0.0600, debugActualVelocityA=-0.0600, debugActualVelocityB=-0.0600, odometer=392282.8700, timestamp=1667506455.6000
      		 
      1667506455.690: motors: raw=[-2453300000.0, -2451200000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
      		 
      1667506455.690: ccwFollowingError: positionError=-0.0100, velocityError=-0.0600, timestamp=1667506455.6000
      		 
      1667506455.690: target: position=-25.5512, velocity=-1.0000, tai=1667506455.6448
      		 
      1667506455.740: rotation: demandPosition=-25.0200, demandVelocity=-0.0600, demandAcceleration=-0.1200, actualPosition=-25.0200, actualVelocity=-0.0600, debugActualVelocityA=-0.0600, debugActualVelocityB=-0.0600, odometer=392282.8700, timestamp=1667506455.6500
      		 
      1667506455.740: motors: raw=[-2453200000.0, -2451100000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
      		 
      1667506455.741: ccwFollowingError: positionError=-0.0200, velocityError=-0.0600, timestamp=1667506455.6500
      		 
      1667506455.790: rotation: demandPosition=-25.0200, demandVelocity=-0.0700, demandAcceleration=-0.1200, actualPosition=-25.0200, actualVelocity=-0.0700, debugActualVelocityA=-0.0700, debugActualVelocityB=-0.0700, odometer=392282.8700, timestamp=1667506455.7000
      		 
      1667506455.790: motors: raw=[-2453100000.0, -2451000000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
      		 
      1667506455.791: ccwFollowingError: positionError=-0.0200, velocityError=-0.0700, timestamp=1667506455.7000
      		 
      track angle=-25.75 velocity=-1.00
      		 
      1667506455.840: rotation: demandPosition=-25.0200, demandVelocity=-0.0800, demandAcceleration=-0.1200, actualPosition=-25.0200, actualVelocity=-0.0800, debugActualVelocityA=-0.0800, debugActualVelocityB=-0.0800, odometer=392282.8800, timestamp=1667506455.7500
      		 
      1667506455.840: motors: raw=[-2452900000.0, -2450800000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
      		 
      1667506455.840: ccwFollowingError: positionError=-0.0200, velocityError=-0.0800, timestamp=1667506455.7500
      		 
      1667506455.890: rotation: demandPosition=-25.0300, demandVelocity=-0.0800, demandAcceleration=-0.1200, actualPosition=-25.0300, actualVelocity=-0.0800, debugActualVelocityA=-0.0800, debugActualVelocityB=-0.0800, odometer=392282.8800, timestamp=1667506455.8000
      		 
      1667506455.890: motors: raw=[-2452800000.0, -2450700000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
      		 
      1667506455.890: ccwFollowingError: positionError=-0.0300, velocityError=-0.0800, timestamp=1667506455.8000
      		 
      1667506455.890: target: position=-25.7517, velocity=-1.0000, tai=1667506455.8453
      		 
      1667506455.940: rotation: demandPosition=-25.0300, demandVelocity=-0.0900, demandAcceleration=-0.1200, actualPosition=-25.0300, actualVelocity=-0.0900, debugActualVelocityA=-0.0900, debugActualVelocityB=-0.0800, odometer=392282.8800, timestamp=1667506455.8500
      		 
      1667506455.940: motors: raw=[-2452600000.0, -2450500000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
      		 
      1667506455.940: ccwFollowingError: positionError=-0.0300, velocityError=-0.0900, timestamp=1667506455.8500
      		 
      1667506455.990: rotation: demandPosition=-25.0400, demandVelocity=-0.0900, demandAcceleration=-0.1200, actualPosition=-25.0400, actualVelocity=-0.0900, debugActualVelocityA=-0.0900, debugActualVelocityB=-0.0900, odometer=392282.8900, timestamp=1667506455.9000
      		 
      1667506455.990: motors: raw=[-2452500000.0, -2450300000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
      		 
      1667506455.990: ccwFollowingError: positionError=-0.0400, velocityError=-0.0900, timestamp=1667506455.9000
      		 
      track angle=-25.95 velocity=-1.00
      		 
      1667506456.040: rotation: demandPosition=-25.0400, demandVelocity=-0.1000, demandAcceleration=-0.1100, actualPosition=-25.0400, actualVelocity=-0.1000, debugActualVelocityA=-0.1000, debugActualVelocityB=-0.1000, odometer=392282.8900, timestamp=1667506455.9500
      		 
      1667506456.040: motors: raw=[-2452300000.0, -2450200000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
      		 
      1667506456.041: ccwFollowingError: positionError=-0.0400, velocityError=-0.1000, timestamp=1667506455.9500
      		 
      1667506456.090: rotation: demandPosition=-25.0500, demandVelocity=-0.1000, demandAcceleration=-0.0500, actualPosition=-25.0500, actualVelocity=-0.1000, debugActualVelocityA=-0.1000, debugActualVelocityB=-0.1100, odometer=392282.9000, timestamp=1667506456.0000
      		 
      1667506456.090: motors: raw=[-2452100000.0, -2450000000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
      		 
      1667506456.090: ccwFollowingError: positionError=-0.0500, velocityError=-0.1100, timestamp=1667506456.0000
      		 
      1667506456.091: target: position=-25.9516, velocity=-1.0000, tai=1667506456.0452
      		 
      1667506456.140: rotation: demandPosition=-25.0500, demandVelocity=-0.1000, demandAcceleration=0.0200, actualPosition=-25.0500, actualVelocity=-0.1000, debugActualVelocityA=-0.1100, debugActualVelocityB=-0.1000, odometer=392282.9000, timestamp=1667506456.0500
      		 
      1667506456.140: motors: raw=[-2451900000.0, -2449800000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
      		 
      1667506456.140: ccwFollowingError: positionError=-0.0500, velocityError=-0.1000, timestamp=1667506456.0500
      		 
      1667506456.190: rotation: demandPosition=-25.0600, demandVelocity=-0.1000, demandAcceleration=0.1600, actualPosition=-25.0600, actualVelocity=-0.1000, debugActualVelocityA=-0.1000, debugActualVelocityB=-0.1000, odometer=392282.9100, timestamp=1667506456.1000
      		 
      1667506456.190: motors: raw=[-2451700000.0, -2449600000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
      		 
      track angle=-26.00 velocity=0.00
      		 
      1667506456.240: rotation: demandPosition=-25.0600, demandVelocity=-0.0900, demandAcceleration=0.3000, actualPosition=-25.0600, actualVelocity=-0.0900, debugActualVelocityA=-0.0900, debugActualVelocityB=-0.1000, odometer=392282.9100, timestamp=1667506456.1500
      		 
      1667506456.240: motors: raw=[-2451500000.0, -2449400000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
      		 
      1667506456.240: ccwFollowingError: positionError=-0.0600, velocityError=-0.0900, timestamp=1667506456.1500
      		 
      1667506456.290: rotation: demandPosition=-25.0600, demandVelocity=-0.0700, demandAcceleration=0.3700, actualPosition=-25.0600, actualVelocity=-0.0700, debugActualVelocityA=-0.0700, debugActualVelocityB=-0.0700, odometer=392282.9200, timestamp=1667506456.2000
      		 
      1667506456.290: motors: raw=[-2451400000.0, -2449200000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
      		 
      1667506456.291: ccwFollowingError: positionError=-0.0600, velocityError=-0.0700, timestamp=1667506456.2000
      		 
      1667506456.290: tracking: tracking=False, lost=True, noNewCommand=False
      		 
      1667506456.291: target: position=-26.0000, velocity=0.0000, tai=1667506456.2452
      		 
      1667506456.290: inPosition: inPosition=False

      This shows an attempt to track a path from -25 to -26 degrees at 1 deg/second (thus a very short ramp). The rotator starts moving in the correct direction but soon turns around. The tracking commands look like this: track angle=-25.95 velocity=-1.00 (though you can also get the same information from the "target: " events, which the CSC publishes when it sends a track command or point-to-point move command to the low-level controller).

      I also attached a graph from Julen Garcia showing position vs time (of both rotator and camera cable wrap). This shows the rotator starting off in the correct direction, but then turning around and ending up in the wrong direction from the starting point. Also notice that the end of the motion is very abrupt; I doubt it is properly jerk-limited.

        Attachments

        1. rotator ramp path.png
          rotator ramp path.png
          261 kB
        2. Rotator PosVelAcc.jpg
          Rotator PosVelAcc.jpg
          193 kB
        3. failed ramp.txt
          235 kB

          Issue Links

          relates to

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-36869 The MT rotator can drive itself past the software limits into the direction inhibit limit swiches

          • To Do

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-25245 Put the Rotator into Fault State if no Track Command for a Long Time

          • Done

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-36872 MT rotator TRACK enabled substate command reports done before the rotator is ready to receive TRACK_VEL_CMD commands

          • Done

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-36860 MTRotator low-level controller

          • Invalid

            Activity

            Ascending order - Click to sort in descending order
            5 older comments
            Hide
            Permalink
            ttsai Te-Wei Tsai added a comment -

            Please help to review the PR:
            https://github.com/lsst-ts/ts_rotator_controller/pull/47

            Thanks!

            Show
            ttsai Te-Wei Tsai added a comment - Please help to review the PR: https://github.com/lsst-ts/ts_rotator_controller/pull/47 Thanks!
            Hide
            Permalink
            rowen Russell Owen added a comment -

            I would not like close this until we can test it. The code from Tekniker that runs on the rotator is similar to the code in the TMA, but not identical.

            Show
            rowen Russell Owen added a comment - I would not like close this until we can test it. The code from Tekniker that runs on the rotator is similar to the code in the TMA, but not identical.
            Hide
            Permalink
            ttsai Te-Wei Tsai added a comment -

            In this case, I would prefer:
            1. You still review that PR because I would like to put the latest Simulink update by Tekniker (which is the same as TMA base on Ismael) to the develop branch. Although I could create another ticket for this, I prefer not.
            2. I put this ticket back to "To Do" since I do not know when the rotator will be recovered for us to test.

            Please help to comment. Thanks!

            Show
            ttsai Te-Wei Tsai added a comment - In this case, I would prefer: 1. You still review that PR because I would like to put the latest Simulink update by Tekniker (which is the same as TMA base on Ismael) to the develop branch. Although I could create another ticket for this, I prefer not. 2. I put this ticket back to "To Do" since I do not know when the rotator will be recovered for us to test. Please help to comment. Thanks!
            Hide
            Permalink
            rowen Russell Owen added a comment -

            Maybe it makes the most sense to close this one and start a new one if we still have problems. I'll review the PR next week when I'm back at work.

            Show
            rowen Russell Owen added a comment - Maybe it makes the most sense to close this one and start a new one if we still have problems. I'll review the PR next week when I'm back at work.
            Hide
            Permalink
            ttsai Te-Wei Tsai added a comment -

            Thanks Russell to review and approve the PR!

            Show
            ttsai Te-Wei Tsai added a comment - Thanks Russell to review and approve the PR!

              People

              Assignee:
              ttsai Te-Wei Tsai
              Reporter:
              rowen Russell Owen
              Reviewers:
              Russell Owen
              Watchers:
              Brian Stalder, Bruno Quint, Julen Garcia, Russell Owen, Te-Wei Tsai
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Jenkins

                  No builds found.