Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-36861

The MT rotator is going in the wrong direction when tracking ramps

    XMLWordPrintable

    Details

      Description

      In the last two days every time we try a ramp tracking sequence for the MT rotator from the MTRotator commander, the rotator moves in the wrong direction, often quite far in the wrong direction. Here is an example, with a very short ramp (so I didn't have to stop it):

      MT Rotator failed ramp 2022-11-03

      ramp -25 -26 1
      		 
      run_command running command method <bound method RotatorCommander.do_ramp of <lsst.ts.mtrotator.rotator_commander.RotatorCommander object at 0x7fc1390205e0>>
      		 
      Tracking a ramp from -25.0 to -26.0 at speed 1.0; this will take 1.00 seconds
      		 
      Finished command ramp
      		 
      track angle=-25.00 velocity=-1.00
      		 
      1667506455.090: controllerState: controllerState=2, offlineSubstate=1, enabledSubstate=2, applicationStatus=5124
      		 
      1667506455.090: tracking: tracking=False, lost=True, noNewCommand=False
      		 
      1667506455.091: target: position=-25.0000, velocity=-1.0000, tai=1667506455.0936
      		 
      1667506455.090: inPosition: inPosition=False
      		 
      track angle=-25.15 velocity=-1.00
      		 
      1667506455.240: rotation: demandPosition=-25.0000, demandVelocity=-0.0000, demandAcceleration=-0.0600, actualPosition=-25.0000, actualVelocity=-0.0000, debugActualVelocityA=-0.0000, debugActualVelocityB=-0.0000, odometer=392282.8500, timestamp=1667506455.1500
      		 
      1667506455.240: tracking: tracking=True, lost=True, noNewCommand=False
      		 
      1667506455.240: inPosition: inPosition=True
      		 
      1667506455.290: rotation: demandPosition=-25.0000, demandVelocity=-0.0100, demandAcceleration=-0.1200, actualPosition=-25.0000, actualVelocity=-0.0000, debugActualVelocityA=-0.0000, debugActualVelocityB=-0.0000, odometer=392282.8500, timestamp=1667506455.2000
      		 
      1667506455.290: target: position=-25.1530, velocity=-1.0000, tai=1667506455.2466
      		 
      1667506455.340: rotation: demandPosition=-25.0000, demandVelocity=-0.0100, demandAcceleration=-0.1200, actualPosition=-25.0000, actualVelocity=-0.0200, debugActualVelocityA=-0.0200, debugActualVelocityB=-0.0100, odometer=392282.8500, timestamp=1667506455.2500
      		 
      1667506455.340: ccwFollowingError: positionError=-0.0000, velocityError=-0.0200, timestamp=1667506455.2500
      		 
      1667506455.390: rotation: demandPosition=-25.0000, demandVelocity=-0.0200, demandAcceleration=-0.1200, actualPosition=-25.0000, actualVelocity=-0.0300, debugActualVelocityA=-0.0300, debugActualVelocityB=-0.0300, odometer=392282.8600, timestamp=1667506455.3000
      		 
      1667506455.390: motors: raw=[-2453700000.0, -2451600000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
      		 
      1667506455.390: ccwFollowingError: positionError=-0.0000, velocityError=-0.0300, timestamp=1667506455.3000
      		 
      track angle=-25.35 velocity=-1.00
      		 
      1667506455.440: rotation: demandPosition=-25.0000, demandVelocity=-0.0300, demandAcceleration=-0.1200, actualPosition=-25.0000, actualVelocity=-0.0300, debugActualVelocityA=-0.0300, debugActualVelocityB=-0.0300, odometer=392282.8600, timestamp=1667506455.3500
      		 
      1667506455.490: motors: raw=[-2453600000.0, -2451500000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
      		 
      1667506455.490: target: position=-25.3514, velocity=-1.0000, tai=1667506455.4450
      		 
      1667506455.540: rotation: demandPosition=-25.0100, demandVelocity=-0.0400, demandAcceleration=-0.1200, actualPosition=-25.0100, actualVelocity=-0.0400, debugActualVelocityA=-0.0400, debugActualVelocityB=-0.0400, odometer=392282.8600, timestamp=1667506455.4500
      		 
      1667506455.540: motors: raw=[-2453600000.0, -2451400000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
      		 
      1667506455.540: ccwFollowingError: positionError=-0.0100, velocityError=-0.0400, timestamp=1667506455.4500
      		 
      1667506455.590: motors: raw=[-2453500000.0, -2451400000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
      		 
      track angle=-25.55 velocity=-1.00
      		 
      1667506455.640: rotation: demandPosition=-25.0100, demandVelocity=-0.0500, demandAcceleration=-0.1200, actualPosition=-25.0100, actualVelocity=-0.0500, debugActualVelocityA=-0.0500, debugActualVelocityB=-0.0500, odometer=392282.8600, timestamp=1667506455.5500
      		 
      1667506455.640: motors: raw=[-2453400000.0, -2451300000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
      		 
      1667506455.640: ccwFollowingError: positionError=-0.0100, velocityError=-0.0500, timestamp=1667506455.5500
      		 
      1667506455.690: rotation: demandPosition=-25.0100, demandVelocity=-0.0600, demandAcceleration=-0.1200, actualPosition=-25.0100, actualVelocity=-0.0600, debugActualVelocityA=-0.0600, debugActualVelocityB=-0.0600, odometer=392282.8700, timestamp=1667506455.6000
      		 
      1667506455.690: motors: raw=[-2453300000.0, -2451200000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
      		 
      1667506455.690: ccwFollowingError: positionError=-0.0100, velocityError=-0.0600, timestamp=1667506455.6000
      		 
      1667506455.690: target: position=-25.5512, velocity=-1.0000, tai=1667506455.6448
      		 
      1667506455.740: rotation: demandPosition=-25.0200, demandVelocity=-0.0600, demandAcceleration=-0.1200, actualPosition=-25.0200, actualVelocity=-0.0600, debugActualVelocityA=-0.0600, debugActualVelocityB=-0.0600, odometer=392282.8700, timestamp=1667506455.6500
      		 
      1667506455.740: motors: raw=[-2453200000.0, -2451100000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
      		 
      1667506455.741: ccwFollowingError: positionError=-0.0200, velocityError=-0.0600, timestamp=1667506455.6500
      		 
      1667506455.790: rotation: demandPosition=-25.0200, demandVelocity=-0.0700, demandAcceleration=-0.1200, actualPosition=-25.0200, actualVelocity=-0.0700, debugActualVelocityA=-0.0700, debugActualVelocityB=-0.0700, odometer=392282.8700, timestamp=1667506455.7000
      		 
      1667506455.790: motors: raw=[-2453100000.0, -2451000000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
      		 
      1667506455.791: ccwFollowingError: positionError=-0.0200, velocityError=-0.0700, timestamp=1667506455.7000
      		 
      track angle=-25.75 velocity=-1.00
      		 
      1667506455.840: rotation: demandPosition=-25.0200, demandVelocity=-0.0800, demandAcceleration=-0.1200, actualPosition=-25.0200, actualVelocity=-0.0800, debugActualVelocityA=-0.0800, debugActualVelocityB=-0.0800, odometer=392282.8800, timestamp=1667506455.7500
      		 
      1667506455.840: motors: raw=[-2452900000.0, -2450800000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
      		 
      1667506455.840: ccwFollowingError: positionError=-0.0200, velocityError=-0.0800, timestamp=1667506455.7500
      		 
      1667506455.890: rotation: demandPosition=-25.0300, demandVelocity=-0.0800, demandAcceleration=-0.1200, actualPosition=-25.0300, actualVelocity=-0.0800, debugActualVelocityA=-0.0800, debugActualVelocityB=-0.0800, odometer=392282.8800, timestamp=1667506455.8000
      		 
      1667506455.890: motors: raw=[-2452800000.0, -2450700000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
      		 
      1667506455.890: ccwFollowingError: positionError=-0.0300, velocityError=-0.0800, timestamp=1667506455.8000
      		 
      1667506455.890: target: position=-25.7517, velocity=-1.0000, tai=1667506455.8453
      		 
      1667506455.940: rotation: demandPosition=-25.0300, demandVelocity=-0.0900, demandAcceleration=-0.1200, actualPosition=-25.0300, actualVelocity=-0.0900, debugActualVelocityA=-0.0900, debugActualVelocityB=-0.0800, odometer=392282.8800, timestamp=1667506455.8500
      		 
      1667506455.940: motors: raw=[-2452600000.0, -2450500000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
      		 
      1667506455.940: ccwFollowingError: positionError=-0.0300, velocityError=-0.0900, timestamp=1667506455.8500
      		 
      1667506455.990: rotation: demandPosition=-25.0400, demandVelocity=-0.0900, demandAcceleration=-0.1200, actualPosition=-25.0400, actualVelocity=-0.0900, debugActualVelocityA=-0.0900, debugActualVelocityB=-0.0900, odometer=392282.8900, timestamp=1667506455.9000
      		 
      1667506455.990: motors: raw=[-2452500000.0, -2450300000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
      		 
      1667506455.990: ccwFollowingError: positionError=-0.0400, velocityError=-0.0900, timestamp=1667506455.9000
      		 
      track angle=-25.95 velocity=-1.00
      		 
      1667506456.040: rotation: demandPosition=-25.0400, demandVelocity=-0.1000, demandAcceleration=-0.1100, actualPosition=-25.0400, actualVelocity=-0.1000, debugActualVelocityA=-0.1000, debugActualVelocityB=-0.1000, odometer=392282.8900, timestamp=1667506455.9500
      		 
      1667506456.040: motors: raw=[-2452300000.0, -2450200000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
      		 
      1667506456.041: ccwFollowingError: positionError=-0.0400, velocityError=-0.1000, timestamp=1667506455.9500
      		 
      1667506456.090: rotation: demandPosition=-25.0500, demandVelocity=-0.1000, demandAcceleration=-0.0500, actualPosition=-25.0500, actualVelocity=-0.1000, debugActualVelocityA=-0.1000, debugActualVelocityB=-0.1100, odometer=392282.9000, timestamp=1667506456.0000
      		 
      1667506456.090: motors: raw=[-2452100000.0, -2450000000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
      		 
      1667506456.090: ccwFollowingError: positionError=-0.0500, velocityError=-0.1100, timestamp=1667506456.0000
      		 
      1667506456.091: target: position=-25.9516, velocity=-1.0000, tai=1667506456.0452
      		 
      1667506456.140: rotation: demandPosition=-25.0500, demandVelocity=-0.1000, demandAcceleration=0.0200, actualPosition=-25.0500, actualVelocity=-0.1000, debugActualVelocityA=-0.1100, debugActualVelocityB=-0.1000, odometer=392282.9000, timestamp=1667506456.0500
      		 
      1667506456.140: motors: raw=[-2451900000.0, -2449800000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
      		 
      1667506456.140: ccwFollowingError: positionError=-0.0500, velocityError=-0.1000, timestamp=1667506456.0500
      		 
      1667506456.190: rotation: demandPosition=-25.0600, demandVelocity=-0.1000, demandAcceleration=0.1600, actualPosition=-25.0600, actualVelocity=-0.1000, debugActualVelocityA=-0.1000, debugActualVelocityB=-0.1000, odometer=392282.9100, timestamp=1667506456.1000
      		 
      1667506456.190: motors: raw=[-2451700000.0, -2449600000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
      		 
      track angle=-26.00 velocity=0.00
      		 
      1667506456.240: rotation: demandPosition=-25.0600, demandVelocity=-0.0900, demandAcceleration=0.3000, actualPosition=-25.0600, actualVelocity=-0.0900, debugActualVelocityA=-0.0900, debugActualVelocityB=-0.1000, odometer=392282.9100, timestamp=1667506456.1500
      		 
      1667506456.240: motors: raw=[-2451500000.0, -2449400000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
      		 
      1667506456.240: ccwFollowingError: positionError=-0.0600, velocityError=-0.0900, timestamp=1667506456.1500
      		 
      1667506456.290: rotation: demandPosition=-25.0600, demandVelocity=-0.0700, demandAcceleration=0.3700, actualPosition=-25.0600, actualVelocity=-0.0700, debugActualVelocityA=-0.0700, debugActualVelocityB=-0.0700, odometer=392282.9200, timestamp=1667506456.2000
      		 
      1667506456.290: motors: raw=[-2451400000.0, -2449200000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
      		 
      1667506456.291: ccwFollowingError: positionError=-0.0600, velocityError=-0.0700, timestamp=1667506456.2000
      		 
      1667506456.290: tracking: tracking=False, lost=True, noNewCommand=False
      		 
      1667506456.291: target: position=-26.0000, velocity=0.0000, tai=1667506456.2452
      		 
      1667506456.290: inPosition: inPosition=False

      This shows an attempt to track a path from -25 to -26 degrees at 1 deg/second (thus a very short ramp). The rotator starts moving in the correct direction but soon turns around. The tracking commands look like this: track angle=-25.95 velocity=-1.00 (though you can also get the same information from the "target: " events, which the CSC publishes when it sends a track command or point-to-point move command to the low-level controller).

      I also attached a graph from Julen Garcia showing position vs time (of both rotator and camera cable wrap). This shows the rotator starting off in the correct direction, but then turning around and ending up in the wrong direction from the starting point. Also notice that the end of the motion is very abrupt; I doubt it is properly jerk-limited.

        Attachments

        1. failed ramp.txt
          235 kB
        2. Rotator PosVelAcc.jpg
          Rotator PosVelAcc.jpg
          193 kB
        3. rotator ramp path.png
          rotator ramp path.png
          261 kB

          Issue Links

          relates to

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-36869 The MT rotator can drive itself past the software limits into the direction inhibit limit swiches

          • To Do

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-25245 Put the Rotator into Fault State if no Track Command for a Long Time

          • Done

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-36872 MT rotator TRACK enabled substate command reports done before the rotator is ready to receive TRACK_VEL_CMD commands

          • Done

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-36860 MTRotator low-level controller

          • Invalid

            Activity

            Ascending order - Click to sort in descending order
            rowen Russell Owen created issue -
            rowen Russell Owen made changes -
            Field Original Value New Value
            Attachment rotator ramp path.png [ 64422 ]
            rowen Russell Owen made changes -
            Description In the last two days every time we try a ramp tracking sequence for the MT rotator from the MTRotator commander, the rotator moves in the wrong direction. Here is an example:

            MT Rotator failed ramp 2022-11-03

            {code}
            ramp -25 -26 1
            run_command running command method <bound method RotatorCommander.do_ramp of <lsst.ts.mtrotator.rotator_commander.RotatorCommander object at 0x7fc1390205e0>>
            Tracking a ramp from -25.0 to -26.0 at speed 1.0; this will take 1.00 seconds
            Finished command ramp
            track angle=-25.00 velocity=-1.00
            1667506455.090: controllerState: controllerState=2, offlineSubstate=1, enabledSubstate=2, applicationStatus=5124
            1667506455.090: tracking: tracking=False, lost=True, noNewCommand=False
            1667506455.091: target: position=-25.0000, velocity=-1.0000, tai=1667506455.0936
            1667506455.090: inPosition: inPosition=False
            track angle=-25.15 velocity=-1.00
            1667506455.240: rotation: demandPosition=-25.0000, demandVelocity=-0.0000, demandAcceleration=-0.0600, actualPosition=-25.0000, actualVelocity=-0.0000, debugActualVelocityA=-0.0000, debugActualVelocityB=-0.0000, odometer=392282.8500, timestamp=1667506455.1500
            1667506455.240: tracking: tracking=True, lost=True, noNewCommand=False
            1667506455.240: inPosition: inPosition=True
            1667506455.290: rotation: demandPosition=-25.0000, demandVelocity=-0.0100, demandAcceleration=-0.1200, actualPosition=-25.0000, actualVelocity=-0.0000, debugActualVelocityA=-0.0000, debugActualVelocityB=-0.0000, odometer=392282.8500, timestamp=1667506455.2000
            1667506455.290: target: position=-25.1530, velocity=-1.0000, tai=1667506455.2466
            1667506455.340: rotation: demandPosition=-25.0000, demandVelocity=-0.0100, demandAcceleration=-0.1200, actualPosition=-25.0000, actualVelocity=-0.0200, debugActualVelocityA=-0.0200, debugActualVelocityB=-0.0100, odometer=392282.8500, timestamp=1667506455.2500
            1667506455.340: ccwFollowingError: positionError=-0.0000, velocityError=-0.0200, timestamp=1667506455.2500
            1667506455.390: rotation: demandPosition=-25.0000, demandVelocity=-0.0200, demandAcceleration=-0.1200, actualPosition=-25.0000, actualVelocity=-0.0300, debugActualVelocityA=-0.0300, debugActualVelocityB=-0.0300, odometer=392282.8600, timestamp=1667506455.3000
            1667506455.390: motors: raw=[-2453700000.0, -2451600000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
            1667506455.390: ccwFollowingError: positionError=-0.0000, velocityError=-0.0300, timestamp=1667506455.3000
            track angle=-25.35 velocity=-1.00
            1667506455.440: rotation: demandPosition=-25.0000, demandVelocity=-0.0300, demandAcceleration=-0.1200, actualPosition=-25.0000, actualVelocity=-0.0300, debugActualVelocityA=-0.0300, debugActualVelocityB=-0.0300, odometer=392282.8600, timestamp=1667506455.3500
            1667506455.490: motors: raw=[-2453600000.0, -2451500000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
            1667506455.490: target: position=-25.3514, velocity=-1.0000, tai=1667506455.4450
            1667506455.540: rotation: demandPosition=-25.0100, demandVelocity=-0.0400, demandAcceleration=-0.1200, actualPosition=-25.0100, actualVelocity=-0.0400, debugActualVelocityA=-0.0400, debugActualVelocityB=-0.0400, odometer=392282.8600, timestamp=1667506455.4500
            1667506455.540: motors: raw=[-2453600000.0, -2451400000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
            1667506455.540: ccwFollowingError: positionError=-0.0100, velocityError=-0.0400, timestamp=1667506455.4500
            1667506455.590: motors: raw=[-2453500000.0, -2451400000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
            track angle=-25.55 velocity=-1.00
            1667506455.640: rotation: demandPosition=-25.0100, demandVelocity=-0.0500, demandAcceleration=-0.1200, actualPosition=-25.0100, actualVelocity=-0.0500, debugActualVelocityA=-0.0500, debugActualVelocityB=-0.0500, odometer=392282.8600, timestamp=1667506455.5500
            1667506455.640: motors: raw=[-2453400000.0, -2451300000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
            1667506455.640: ccwFollowingError: positionError=-0.0100, velocityError=-0.0500, timestamp=1667506455.5500
            1667506455.690: rotation: demandPosition=-25.0100, demandVelocity=-0.0600, demandAcceleration=-0.1200, actualPosition=-25.0100, actualVelocity=-0.0600, debugActualVelocityA=-0.0600, debugActualVelocityB=-0.0600, odometer=392282.8700, timestamp=1667506455.6000
            1667506455.690: motors: raw=[-2453300000.0, -2451200000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
            1667506455.690: ccwFollowingError: positionError=-0.0100, velocityError=-0.0600, timestamp=1667506455.6000
            1667506455.690: target: position=-25.5512, velocity=-1.0000, tai=1667506455.6448
            1667506455.740: rotation: demandPosition=-25.0200, demandVelocity=-0.0600, demandAcceleration=-0.1200, actualPosition=-25.0200, actualVelocity=-0.0600, debugActualVelocityA=-0.0600, debugActualVelocityB=-0.0600, odometer=392282.8700, timestamp=1667506455.6500
            1667506455.740: motors: raw=[-2453200000.0, -2451100000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
            1667506455.741: ccwFollowingError: positionError=-0.0200, velocityError=-0.0600, timestamp=1667506455.6500
            1667506455.790: rotation: demandPosition=-25.0200, demandVelocity=-0.0700, demandAcceleration=-0.1200, actualPosition=-25.0200, actualVelocity=-0.0700, debugActualVelocityA=-0.0700, debugActualVelocityB=-0.0700, odometer=392282.8700, timestamp=1667506455.7000
            1667506455.790: motors: raw=[-2453100000.0, -2451000000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
            1667506455.791: ccwFollowingError: positionError=-0.0200, velocityError=-0.0700, timestamp=1667506455.7000
            track angle=-25.75 velocity=-1.00
            1667506455.840: rotation: demandPosition=-25.0200, demandVelocity=-0.0800, demandAcceleration=-0.1200, actualPosition=-25.0200, actualVelocity=-0.0800, debugActualVelocityA=-0.0800, debugActualVelocityB=-0.0800, odometer=392282.8800, timestamp=1667506455.7500
            1667506455.840: motors: raw=[-2452900000.0, -2450800000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
            1667506455.840: ccwFollowingError: positionError=-0.0200, velocityError=-0.0800, timestamp=1667506455.7500
            1667506455.890: rotation: demandPosition=-25.0300, demandVelocity=-0.0800, demandAcceleration=-0.1200, actualPosition=-25.0300, actualVelocity=-0.0800, debugActualVelocityA=-0.0800, debugActualVelocityB=-0.0800, odometer=392282.8800, timestamp=1667506455.8000
            1667506455.890: motors: raw=[-2452800000.0, -2450700000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
            1667506455.890: ccwFollowingError: positionError=-0.0300, velocityError=-0.0800, timestamp=1667506455.8000
            1667506455.890: target: position=-25.7517, velocity=-1.0000, tai=1667506455.8453
            1667506455.940: rotation: demandPosition=-25.0300, demandVelocity=-0.0900, demandAcceleration=-0.1200, actualPosition=-25.0300, actualVelocity=-0.0900, debugActualVelocityA=-0.0900, debugActualVelocityB=-0.0800, odometer=392282.8800, timestamp=1667506455.8500
            1667506455.940: motors: raw=[-2452600000.0, -2450500000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
            1667506455.940: ccwFollowingError: positionError=-0.0300, velocityError=-0.0900, timestamp=1667506455.8500
            1667506455.990: rotation: demandPosition=-25.0400, demandVelocity=-0.0900, demandAcceleration=-0.1200, actualPosition=-25.0400, actualVelocity=-0.0900, debugActualVelocityA=-0.0900, debugActualVelocityB=-0.0900, odometer=392282.8900, timestamp=1667506455.9000
            1667506455.990: motors: raw=[-2452500000.0, -2450300000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
            1667506455.990: ccwFollowingError: positionError=-0.0400, velocityError=-0.0900, timestamp=1667506455.9000
            track angle=-25.95 velocity=-1.00
            1667506456.040: rotation: demandPosition=-25.0400, demandVelocity=-0.1000, demandAcceleration=-0.1100, actualPosition=-25.0400, actualVelocity=-0.1000, debugActualVelocityA=-0.1000, debugActualVelocityB=-0.1000, odometer=392282.8900, timestamp=1667506455.9500
            1667506456.040: motors: raw=[-2452300000.0, -2450200000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
            1667506456.041: ccwFollowingError: positionError=-0.0400, velocityError=-0.1000, timestamp=1667506455.9500
            1667506456.090: rotation: demandPosition=-25.0500, demandVelocity=-0.1000, demandAcceleration=-0.0500, actualPosition=-25.0500, actualVelocity=-0.1000, debugActualVelocityA=-0.1000, debugActualVelocityB=-0.1100, odometer=392282.9000, timestamp=1667506456.0000
            1667506456.090: motors: raw=[-2452100000.0, -2450000000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
            1667506456.090: ccwFollowingError: positionError=-0.0500, velocityError=-0.1100, timestamp=1667506456.0000
            1667506456.091: target: position=-25.9516, velocity=-1.0000, tai=1667506456.0452
            1667506456.140: rotation: demandPosition=-25.0500, demandVelocity=-0.1000, demandAcceleration=0.0200, actualPosition=-25.0500, actualVelocity=-0.1000, debugActualVelocityA=-0.1100, debugActualVelocityB=-0.1000, odometer=392282.9000, timestamp=1667506456.0500
            1667506456.140: motors: raw=[-2451900000.0, -2449800000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
            1667506456.140: ccwFollowingError: positionError=-0.0500, velocityError=-0.1000, timestamp=1667506456.0500
            1667506456.190: rotation: demandPosition=-25.0600, demandVelocity=-0.1000, demandAcceleration=0.1600, actualPosition=-25.0600, actualVelocity=-0.1000, debugActualVelocityA=-0.1000, debugActualVelocityB=-0.1000, odometer=392282.9100, timestamp=1667506456.1000
            1667506456.190: motors: raw=[-2451700000.0, -2449600000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
            track angle=-26.00 velocity=0.00
            1667506456.240: rotation: demandPosition=-25.0600, demandVelocity=-0.0900, demandAcceleration=0.3000, actualPosition=-25.0600, actualVelocity=-0.0900, debugActualVelocityA=-0.0900, debugActualVelocityB=-0.1000, odometer=392282.9100, timestamp=1667506456.1500
            1667506456.240: motors: raw=[-2451500000.0, -2449400000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
            1667506456.240: ccwFollowingError: positionError=-0.0600, velocityError=-0.0900, timestamp=1667506456.1500
            1667506456.290: rotation: demandPosition=-25.0600, demandVelocity=-0.0700, demandAcceleration=0.3700, actualPosition=-25.0600, actualVelocity=-0.0700, debugActualVelocityA=-0.0700, debugActualVelocityB=-0.0700, odometer=392282.9200, timestamp=1667506456.2000
            1667506456.290: motors: raw=[-2451400000.0, -2449200000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
            1667506456.291: ccwFollowingError: positionError=-0.0600, velocityError=-0.0700, timestamp=1667506456.2000
            1667506456.290: tracking: tracking=False, lost=True, noNewCommand=False
            1667506456.291: target: position=-26.0000, velocity=0.0000, tai=1667506456.2452
            1667506456.290: inPosition: inPosition=False
            {code}

            This shows an attempt to track a path from -25 to -26 degrees at 1 deg/second (thus a very short ramp). The rotator starts moving in the correct direction but soon turns around. The tracking commands look like this: {{track angle=-25.95 velocity=-1.00}}.

            I will also try to attach a graph from [~julen.garcia] showing position vs time (of both rotator and camera cable wrap).             		In the last two days every time we try a ramp tracking sequence for the MT rotator from the MTRotator commander, the rotator moves in the wrong direction, often quite far in the wrong direction. Here is an example, with a very short ramp (so I didn't have to stop it):

            MT Rotator failed ramp 2022-11-03

            {code}
            ramp -25 -26 1
            run_command running command method <bound method RotatorCommander.do_ramp of <lsst.ts.mtrotator.rotator_commander.RotatorCommander object at 0x7fc1390205e0>>
            Tracking a ramp from -25.0 to -26.0 at speed 1.0; this will take 1.00 seconds
            Finished command ramp
            track angle=-25.00 velocity=-1.00
            1667506455.090: controllerState: controllerState=2, offlineSubstate=1, enabledSubstate=2, applicationStatus=5124
            1667506455.090: tracking: tracking=False, lost=True, noNewCommand=False
            1667506455.091: target: position=-25.0000, velocity=-1.0000, tai=1667506455.0936
            1667506455.090: inPosition: inPosition=False
            track angle=-25.15 velocity=-1.00
            1667506455.240: rotation: demandPosition=-25.0000, demandVelocity=-0.0000, demandAcceleration=-0.0600, actualPosition=-25.0000, actualVelocity=-0.0000, debugActualVelocityA=-0.0000, debugActualVelocityB=-0.0000, odometer=392282.8500, timestamp=1667506455.1500
            1667506455.240: tracking: tracking=True, lost=True, noNewCommand=False
            1667506455.240: inPosition: inPosition=True
            1667506455.290: rotation: demandPosition=-25.0000, demandVelocity=-0.0100, demandAcceleration=-0.1200, actualPosition=-25.0000, actualVelocity=-0.0000, debugActualVelocityA=-0.0000, debugActualVelocityB=-0.0000, odometer=392282.8500, timestamp=1667506455.2000
            1667506455.290: target: position=-25.1530, velocity=-1.0000, tai=1667506455.2466
            1667506455.340: rotation: demandPosition=-25.0000, demandVelocity=-0.0100, demandAcceleration=-0.1200, actualPosition=-25.0000, actualVelocity=-0.0200, debugActualVelocityA=-0.0200, debugActualVelocityB=-0.0100, odometer=392282.8500, timestamp=1667506455.2500
            1667506455.340: ccwFollowingError: positionError=-0.0000, velocityError=-0.0200, timestamp=1667506455.2500
            1667506455.390: rotation: demandPosition=-25.0000, demandVelocity=-0.0200, demandAcceleration=-0.1200, actualPosition=-25.0000, actualVelocity=-0.0300, debugActualVelocityA=-0.0300, debugActualVelocityB=-0.0300, odometer=392282.8600, timestamp=1667506455.3000
            1667506455.390: motors: raw=[-2453700000.0, -2451600000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
            1667506455.390: ccwFollowingError: positionError=-0.0000, velocityError=-0.0300, timestamp=1667506455.3000
            track angle=-25.35 velocity=-1.00
            1667506455.440: rotation: demandPosition=-25.0000, demandVelocity=-0.0300, demandAcceleration=-0.1200, actualPosition=-25.0000, actualVelocity=-0.0300, debugActualVelocityA=-0.0300, debugActualVelocityB=-0.0300, odometer=392282.8600, timestamp=1667506455.3500
            1667506455.490: motors: raw=[-2453600000.0, -2451500000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
            1667506455.490: target: position=-25.3514, velocity=-1.0000, tai=1667506455.4450
            1667506455.540: rotation: demandPosition=-25.0100, demandVelocity=-0.0400, demandAcceleration=-0.1200, actualPosition=-25.0100, actualVelocity=-0.0400, debugActualVelocityA=-0.0400, debugActualVelocityB=-0.0400, odometer=392282.8600, timestamp=1667506455.4500
            1667506455.540: motors: raw=[-2453600000.0, -2451400000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
            1667506455.540: ccwFollowingError: positionError=-0.0100, velocityError=-0.0400, timestamp=1667506455.4500
            1667506455.590: motors: raw=[-2453500000.0, -2451400000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
            track angle=-25.55 velocity=-1.00
            1667506455.640: rotation: demandPosition=-25.0100, demandVelocity=-0.0500, demandAcceleration=-0.1200, actualPosition=-25.0100, actualVelocity=-0.0500, debugActualVelocityA=-0.0500, debugActualVelocityB=-0.0500, odometer=392282.8600, timestamp=1667506455.5500
            1667506455.640: motors: raw=[-2453400000.0, -2451300000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
            1667506455.640: ccwFollowingError: positionError=-0.0100, velocityError=-0.0500, timestamp=1667506455.5500
            1667506455.690: rotation: demandPosition=-25.0100, demandVelocity=-0.0600, demandAcceleration=-0.1200, actualPosition=-25.0100, actualVelocity=-0.0600, debugActualVelocityA=-0.0600, debugActualVelocityB=-0.0600, odometer=392282.8700, timestamp=1667506455.6000
            1667506455.690: motors: raw=[-2453300000.0, -2451200000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
            1667506455.690: ccwFollowingError: positionError=-0.0100, velocityError=-0.0600, timestamp=1667506455.6000
            1667506455.690: target: position=-25.5512, velocity=-1.0000, tai=1667506455.6448
            1667506455.740: rotation: demandPosition=-25.0200, demandVelocity=-0.0600, demandAcceleration=-0.1200, actualPosition=-25.0200, actualVelocity=-0.0600, debugActualVelocityA=-0.0600, debugActualVelocityB=-0.0600, odometer=392282.8700, timestamp=1667506455.6500
            1667506455.740: motors: raw=[-2453200000.0, -2451100000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
            1667506455.741: ccwFollowingError: positionError=-0.0200, velocityError=-0.0600, timestamp=1667506455.6500
            1667506455.790: rotation: demandPosition=-25.0200, demandVelocity=-0.0700, demandAcceleration=-0.1200, actualPosition=-25.0200, actualVelocity=-0.0700, debugActualVelocityA=-0.0700, debugActualVelocityB=-0.0700, odometer=392282.8700, timestamp=1667506455.7000
            1667506455.790: motors: raw=[-2453100000.0, -2451000000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
            1667506455.791: ccwFollowingError: positionError=-0.0200, velocityError=-0.0700, timestamp=1667506455.7000
            track angle=-25.75 velocity=-1.00
            1667506455.840: rotation: demandPosition=-25.0200, demandVelocity=-0.0800, demandAcceleration=-0.1200, actualPosition=-25.0200, actualVelocity=-0.0800, debugActualVelocityA=-0.0800, debugActualVelocityB=-0.0800, odometer=392282.8800, timestamp=1667506455.7500
            1667506455.840: motors: raw=[-2452900000.0, -2450800000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
            1667506455.840: ccwFollowingError: positionError=-0.0200, velocityError=-0.0800, timestamp=1667506455.7500
            1667506455.890: rotation: demandPosition=-25.0300, demandVelocity=-0.0800, demandAcceleration=-0.1200, actualPosition=-25.0300, actualVelocity=-0.0800, debugActualVelocityA=-0.0800, debugActualVelocityB=-0.0800, odometer=392282.8800, timestamp=1667506455.8000
            1667506455.890: motors: raw=[-2452800000.0, -2450700000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
            1667506455.890: ccwFollowingError: positionError=-0.0300, velocityError=-0.0800, timestamp=1667506455.8000
            1667506455.890: target: position=-25.7517, velocity=-1.0000, tai=1667506455.8453
            1667506455.940: rotation: demandPosition=-25.0300, demandVelocity=-0.0900, demandAcceleration=-0.1200, actualPosition=-25.0300, actualVelocity=-0.0900, debugActualVelocityA=-0.0900, debugActualVelocityB=-0.0800, odometer=392282.8800, timestamp=1667506455.8500
            1667506455.940: motors: raw=[-2452600000.0, -2450500000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
            1667506455.940: ccwFollowingError: positionError=-0.0300, velocityError=-0.0900, timestamp=1667506455.8500
            1667506455.990: rotation: demandPosition=-25.0400, demandVelocity=-0.0900, demandAcceleration=-0.1200, actualPosition=-25.0400, actualVelocity=-0.0900, debugActualVelocityA=-0.0900, debugActualVelocityB=-0.0900, odometer=392282.8900, timestamp=1667506455.9000
            1667506455.990: motors: raw=[-2452500000.0, -2450300000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
            1667506455.990: ccwFollowingError: positionError=-0.0400, velocityError=-0.0900, timestamp=1667506455.9000
            track angle=-25.95 velocity=-1.00
            1667506456.040: rotation: demandPosition=-25.0400, demandVelocity=-0.1000, demandAcceleration=-0.1100, actualPosition=-25.0400, actualVelocity=-0.1000, debugActualVelocityA=-0.1000, debugActualVelocityB=-0.1000, odometer=392282.8900, timestamp=1667506455.9500
            1667506456.040: motors: raw=[-2452300000.0, -2450200000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
            1667506456.041: ccwFollowingError: positionError=-0.0400, velocityError=-0.1000, timestamp=1667506455.9500
            1667506456.090: rotation: demandPosition=-25.0500, demandVelocity=-0.1000, demandAcceleration=-0.0500, actualPosition=-25.0500, actualVelocity=-0.1000, debugActualVelocityA=-0.1000, debugActualVelocityB=-0.1100, odometer=392282.9000, timestamp=1667506456.0000
            1667506456.090: motors: raw=[-2452100000.0, -2450000000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
            1667506456.090: ccwFollowingError: positionError=-0.0500, velocityError=-0.1100, timestamp=1667506456.0000
            1667506456.091: target: position=-25.9516, velocity=-1.0000, tai=1667506456.0452
            1667506456.140: rotation: demandPosition=-25.0500, demandVelocity=-0.1000, demandAcceleration=0.0200, actualPosition=-25.0500, actualVelocity=-0.1000, debugActualVelocityA=-0.1100, debugActualVelocityB=-0.1000, odometer=392282.9000, timestamp=1667506456.0500
            1667506456.140: motors: raw=[-2451900000.0, -2449800000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
            1667506456.140: ccwFollowingError: positionError=-0.0500, velocityError=-0.1000, timestamp=1667506456.0500
            1667506456.190: rotation: demandPosition=-25.0600, demandVelocity=-0.1000, demandAcceleration=0.1600, actualPosition=-25.0600, actualVelocity=-0.1000, debugActualVelocityA=-0.1000, debugActualVelocityB=-0.1000, odometer=392282.9100, timestamp=1667506456.1000
            1667506456.190: motors: raw=[-2451700000.0, -2449600000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
            track angle=-26.00 velocity=0.00
            1667506456.240: rotation: demandPosition=-25.0600, demandVelocity=-0.0900, demandAcceleration=0.3000, actualPosition=-25.0600, actualVelocity=-0.0900, debugActualVelocityA=-0.0900, debugActualVelocityB=-0.1000, odometer=392282.9100, timestamp=1667506456.1500
            1667506456.240: motors: raw=[-2451500000.0, -2449400000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
            1667506456.240: ccwFollowingError: positionError=-0.0600, velocityError=-0.0900, timestamp=1667506456.1500
            1667506456.290: rotation: demandPosition=-25.0600, demandVelocity=-0.0700, demandAcceleration=0.3700, actualPosition=-25.0600, actualVelocity=-0.0700, debugActualVelocityA=-0.0700, debugActualVelocityB=-0.0700, odometer=392282.9200, timestamp=1667506456.2000
            1667506456.290: motors: raw=[-2451400000.0, -2449200000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
            1667506456.291: ccwFollowingError: positionError=-0.0600, velocityError=-0.0700, timestamp=1667506456.2000
            1667506456.290: tracking: tracking=False, lost=True, noNewCommand=False
            1667506456.291: target: position=-26.0000, velocity=0.0000, tai=1667506456.2452
            1667506456.290: inPosition: inPosition=False
            {code}

            This shows an attempt to track a path from -25 to -26 degrees at 1 deg/second (thus a very short ramp). The rotator starts moving in the correct direction but soon turns around. The tracking commands look like this: {{track angle=-25.95 velocity=-1.00}}.

            I will also try to attach a graph from [~julen.garcia] showing position vs time (of both rotator and camera cable wrap).
            ttsai Te-Wei Tsai made changes -
            Epic Link DM-27721 [ 442128 ]
            Labels Rotator
            ttsai Te-Wei Tsai made changes -
            Link This issue relates to DM-36860 [ DM-36860 ]
            bquint Bruno Quint made changes -
            Link This issue relates to FRACAS-129 [ FRACAS-129 ]
            bquint Bruno Quint made changes -
            Link This issue relates to FRACAS-129 [ FRACAS-129 ]
            bquint Bruno Quint made changes -
            Link This issue relates to FRACAS-129 [ FRACAS-129 ]
            rowen Russell Owen made changes -
            Description In the last two days every time we try a ramp tracking sequence for the MT rotator from the MTRotator commander, the rotator moves in the wrong direction, often quite far in the wrong direction. Here is an example, with a very short ramp (so I didn't have to stop it):

            MT Rotator failed ramp 2022-11-03

            {code}
            ramp -25 -26 1
            run_command running command method <bound method RotatorCommander.do_ramp of <lsst.ts.mtrotator.rotator_commander.RotatorCommander object at 0x7fc1390205e0>>
            Tracking a ramp from -25.0 to -26.0 at speed 1.0; this will take 1.00 seconds
            Finished command ramp
            track angle=-25.00 velocity=-1.00
            1667506455.090: controllerState: controllerState=2, offlineSubstate=1, enabledSubstate=2, applicationStatus=5124
            1667506455.090: tracking: tracking=False, lost=True, noNewCommand=False
            1667506455.091: target: position=-25.0000, velocity=-1.0000, tai=1667506455.0936
            1667506455.090: inPosition: inPosition=False
            track angle=-25.15 velocity=-1.00
            1667506455.240: rotation: demandPosition=-25.0000, demandVelocity=-0.0000, demandAcceleration=-0.0600, actualPosition=-25.0000, actualVelocity=-0.0000, debugActualVelocityA=-0.0000, debugActualVelocityB=-0.0000, odometer=392282.8500, timestamp=1667506455.1500
            1667506455.240: tracking: tracking=True, lost=True, noNewCommand=False
            1667506455.240: inPosition: inPosition=True
            1667506455.290: rotation: demandPosition=-25.0000, demandVelocity=-0.0100, demandAcceleration=-0.1200, actualPosition=-25.0000, actualVelocity=-0.0000, debugActualVelocityA=-0.0000, debugActualVelocityB=-0.0000, odometer=392282.8500, timestamp=1667506455.2000
            1667506455.290: target: position=-25.1530, velocity=-1.0000, tai=1667506455.2466
            1667506455.340: rotation: demandPosition=-25.0000, demandVelocity=-0.0100, demandAcceleration=-0.1200, actualPosition=-25.0000, actualVelocity=-0.0200, debugActualVelocityA=-0.0200, debugActualVelocityB=-0.0100, odometer=392282.8500, timestamp=1667506455.2500
            1667506455.340: ccwFollowingError: positionError=-0.0000, velocityError=-0.0200, timestamp=1667506455.2500
            1667506455.390: rotation: demandPosition=-25.0000, demandVelocity=-0.0200, demandAcceleration=-0.1200, actualPosition=-25.0000, actualVelocity=-0.0300, debugActualVelocityA=-0.0300, debugActualVelocityB=-0.0300, odometer=392282.8600, timestamp=1667506455.3000
            1667506455.390: motors: raw=[-2453700000.0, -2451600000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
            1667506455.390: ccwFollowingError: positionError=-0.0000, velocityError=-0.0300, timestamp=1667506455.3000
            track angle=-25.35 velocity=-1.00
            1667506455.440: rotation: demandPosition=-25.0000, demandVelocity=-0.0300, demandAcceleration=-0.1200, actualPosition=-25.0000, actualVelocity=-0.0300, debugActualVelocityA=-0.0300, debugActualVelocityB=-0.0300, odometer=392282.8600, timestamp=1667506455.3500
            1667506455.490: motors: raw=[-2453600000.0, -2451500000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
            1667506455.490: target: position=-25.3514, velocity=-1.0000, tai=1667506455.4450
            1667506455.540: rotation: demandPosition=-25.0100, demandVelocity=-0.0400, demandAcceleration=-0.1200, actualPosition=-25.0100, actualVelocity=-0.0400, debugActualVelocityA=-0.0400, debugActualVelocityB=-0.0400, odometer=392282.8600, timestamp=1667506455.4500
            1667506455.540: motors: raw=[-2453600000.0, -2451400000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
            1667506455.540: ccwFollowingError: positionError=-0.0100, velocityError=-0.0400, timestamp=1667506455.4500
            1667506455.590: motors: raw=[-2453500000.0, -2451400000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
            track angle=-25.55 velocity=-1.00
            1667506455.640: rotation: demandPosition=-25.0100, demandVelocity=-0.0500, demandAcceleration=-0.1200, actualPosition=-25.0100, actualVelocity=-0.0500, debugActualVelocityA=-0.0500, debugActualVelocityB=-0.0500, odometer=392282.8600, timestamp=1667506455.5500
            1667506455.640: motors: raw=[-2453400000.0, -2451300000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
            1667506455.640: ccwFollowingError: positionError=-0.0100, velocityError=-0.0500, timestamp=1667506455.5500
            1667506455.690: rotation: demandPosition=-25.0100, demandVelocity=-0.0600, demandAcceleration=-0.1200, actualPosition=-25.0100, actualVelocity=-0.0600, debugActualVelocityA=-0.0600, debugActualVelocityB=-0.0600, odometer=392282.8700, timestamp=1667506455.6000
            1667506455.690: motors: raw=[-2453300000.0, -2451200000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
            1667506455.690: ccwFollowingError: positionError=-0.0100, velocityError=-0.0600, timestamp=1667506455.6000
            1667506455.690: target: position=-25.5512, velocity=-1.0000, tai=1667506455.6448
            1667506455.740: rotation: demandPosition=-25.0200, demandVelocity=-0.0600, demandAcceleration=-0.1200, actualPosition=-25.0200, actualVelocity=-0.0600, debugActualVelocityA=-0.0600, debugActualVelocityB=-0.0600, odometer=392282.8700, timestamp=1667506455.6500
            1667506455.740: motors: raw=[-2453200000.0, -2451100000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
            1667506455.741: ccwFollowingError: positionError=-0.0200, velocityError=-0.0600, timestamp=1667506455.6500
            1667506455.790: rotation: demandPosition=-25.0200, demandVelocity=-0.0700, demandAcceleration=-0.1200, actualPosition=-25.0200, actualVelocity=-0.0700, debugActualVelocityA=-0.0700, debugActualVelocityB=-0.0700, odometer=392282.8700, timestamp=1667506455.7000
            1667506455.790: motors: raw=[-2453100000.0, -2451000000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
            1667506455.791: ccwFollowingError: positionError=-0.0200, velocityError=-0.0700, timestamp=1667506455.7000
            track angle=-25.75 velocity=-1.00
            1667506455.840: rotation: demandPosition=-25.0200, demandVelocity=-0.0800, demandAcceleration=-0.1200, actualPosition=-25.0200, actualVelocity=-0.0800, debugActualVelocityA=-0.0800, debugActualVelocityB=-0.0800, odometer=392282.8800, timestamp=1667506455.7500
            1667506455.840: motors: raw=[-2452900000.0, -2450800000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
            1667506455.840: ccwFollowingError: positionError=-0.0200, velocityError=-0.0800, timestamp=1667506455.7500
            1667506455.890: rotation: demandPosition=-25.0300, demandVelocity=-0.0800, demandAcceleration=-0.1200, actualPosition=-25.0300, actualVelocity=-0.0800, debugActualVelocityA=-0.0800, debugActualVelocityB=-0.0800, odometer=392282.8800, timestamp=1667506455.8000
            1667506455.890: motors: raw=[-2452800000.0, -2450700000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
            1667506455.890: ccwFollowingError: positionError=-0.0300, velocityError=-0.0800, timestamp=1667506455.8000
            1667506455.890: target: position=-25.7517, velocity=-1.0000, tai=1667506455.8453
            1667506455.940: rotation: demandPosition=-25.0300, demandVelocity=-0.0900, demandAcceleration=-0.1200, actualPosition=-25.0300, actualVelocity=-0.0900, debugActualVelocityA=-0.0900, debugActualVelocityB=-0.0800, odometer=392282.8800, timestamp=1667506455.8500
            1667506455.940: motors: raw=[-2452600000.0, -2450500000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
            1667506455.940: ccwFollowingError: positionError=-0.0300, velocityError=-0.0900, timestamp=1667506455.8500
            1667506455.990: rotation: demandPosition=-25.0400, demandVelocity=-0.0900, demandAcceleration=-0.1200, actualPosition=-25.0400, actualVelocity=-0.0900, debugActualVelocityA=-0.0900, debugActualVelocityB=-0.0900, odometer=392282.8900, timestamp=1667506455.9000
            1667506455.990: motors: raw=[-2452500000.0, -2450300000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
            1667506455.990: ccwFollowingError: positionError=-0.0400, velocityError=-0.0900, timestamp=1667506455.9000
            track angle=-25.95 velocity=-1.00
            1667506456.040: rotation: demandPosition=-25.0400, demandVelocity=-0.1000, demandAcceleration=-0.1100, actualPosition=-25.0400, actualVelocity=-0.1000, debugActualVelocityA=-0.1000, debugActualVelocityB=-0.1000, odometer=392282.8900, timestamp=1667506455.9500
            1667506456.040: motors: raw=[-2452300000.0, -2450200000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
            1667506456.041: ccwFollowingError: positionError=-0.0400, velocityError=-0.1000, timestamp=1667506455.9500
            1667506456.090: rotation: demandPosition=-25.0500, demandVelocity=-0.1000, demandAcceleration=-0.0500, actualPosition=-25.0500, actualVelocity=-0.1000, debugActualVelocityA=-0.1000, debugActualVelocityB=-0.1100, odometer=392282.9000, timestamp=1667506456.0000
            1667506456.090: motors: raw=[-2452100000.0, -2450000000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
            1667506456.090: ccwFollowingError: positionError=-0.0500, velocityError=-0.1100, timestamp=1667506456.0000
            1667506456.091: target: position=-25.9516, velocity=-1.0000, tai=1667506456.0452
            1667506456.140: rotation: demandPosition=-25.0500, demandVelocity=-0.1000, demandAcceleration=0.0200, actualPosition=-25.0500, actualVelocity=-0.1000, debugActualVelocityA=-0.1100, debugActualVelocityB=-0.1000, odometer=392282.9000, timestamp=1667506456.0500
            1667506456.140: motors: raw=[-2451900000.0, -2449800000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
            1667506456.140: ccwFollowingError: positionError=-0.0500, velocityError=-0.1000, timestamp=1667506456.0500
            1667506456.190: rotation: demandPosition=-25.0600, demandVelocity=-0.1000, demandAcceleration=0.1600, actualPosition=-25.0600, actualVelocity=-0.1000, debugActualVelocityA=-0.1000, debugActualVelocityB=-0.1000, odometer=392282.9100, timestamp=1667506456.1000
            1667506456.190: motors: raw=[-2451700000.0, -2449600000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
            track angle=-26.00 velocity=0.00
            1667506456.240: rotation: demandPosition=-25.0600, demandVelocity=-0.0900, demandAcceleration=0.3000, actualPosition=-25.0600, actualVelocity=-0.0900, debugActualVelocityA=-0.0900, debugActualVelocityB=-0.1000, odometer=392282.9100, timestamp=1667506456.1500
            1667506456.240: motors: raw=[-2451500000.0, -2449400000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
            1667506456.240: ccwFollowingError: positionError=-0.0600, velocityError=-0.0900, timestamp=1667506456.1500
            1667506456.290: rotation: demandPosition=-25.0600, demandVelocity=-0.0700, demandAcceleration=0.3700, actualPosition=-25.0600, actualVelocity=-0.0700, debugActualVelocityA=-0.0700, debugActualVelocityB=-0.0700, odometer=392282.9200, timestamp=1667506456.2000
            1667506456.290: motors: raw=[-2451400000.0, -2449200000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
            1667506456.291: ccwFollowingError: positionError=-0.0600, velocityError=-0.0700, timestamp=1667506456.2000
            1667506456.290: tracking: tracking=False, lost=True, noNewCommand=False
            1667506456.291: target: position=-26.0000, velocity=0.0000, tai=1667506456.2452
            1667506456.290: inPosition: inPosition=False
            {code}

            This shows an attempt to track a path from -25 to -26 degrees at 1 deg/second (thus a very short ramp). The rotator starts moving in the correct direction but soon turns around. The tracking commands look like this: {{track angle=-25.95 velocity=-1.00}}.

            I will also try to attach a graph from [~julen.garcia] showing position vs time (of both rotator and camera cable wrap).             		In the last two days every time we try a ramp tracking sequence for the MT rotator from the MTRotator commander, the rotator moves in the wrong direction, often quite far in the wrong direction. Here is an example, with a very short ramp (so I didn't have to stop it):

            MT Rotator failed ramp 2022-11-03

            {code}
            ramp -25 -26 1
            run_command running command method <bound method RotatorCommander.do_ramp of <lsst.ts.mtrotator.rotator_commander.RotatorCommander object at 0x7fc1390205e0>>
            Tracking a ramp from -25.0 to -26.0 at speed 1.0; this will take 1.00 seconds
            Finished command ramp
            track angle=-25.00 velocity=-1.00
            1667506455.090: controllerState: controllerState=2, offlineSubstate=1, enabledSubstate=2, applicationStatus=5124
            1667506455.090: tracking: tracking=False, lost=True, noNewCommand=False
            1667506455.091: target: position=-25.0000, velocity=-1.0000, tai=1667506455.0936
            1667506455.090: inPosition: inPosition=False
            track angle=-25.15 velocity=-1.00
            1667506455.240: rotation: demandPosition=-25.0000, demandVelocity=-0.0000, demandAcceleration=-0.0600, actualPosition=-25.0000, actualVelocity=-0.0000, debugActualVelocityA=-0.0000, debugActualVelocityB=-0.0000, odometer=392282.8500, timestamp=1667506455.1500
            1667506455.240: tracking: tracking=True, lost=True, noNewCommand=False
            1667506455.240: inPosition: inPosition=True
            1667506455.290: rotation: demandPosition=-25.0000, demandVelocity=-0.0100, demandAcceleration=-0.1200, actualPosition=-25.0000, actualVelocity=-0.0000, debugActualVelocityA=-0.0000, debugActualVelocityB=-0.0000, odometer=392282.8500, timestamp=1667506455.2000
            1667506455.290: target: position=-25.1530, velocity=-1.0000, tai=1667506455.2466
            1667506455.340: rotation: demandPosition=-25.0000, demandVelocity=-0.0100, demandAcceleration=-0.1200, actualPosition=-25.0000, actualVelocity=-0.0200, debugActualVelocityA=-0.0200, debugActualVelocityB=-0.0100, odometer=392282.8500, timestamp=1667506455.2500
            1667506455.340: ccwFollowingError: positionError=-0.0000, velocityError=-0.0200, timestamp=1667506455.2500
            1667506455.390: rotation: demandPosition=-25.0000, demandVelocity=-0.0200, demandAcceleration=-0.1200, actualPosition=-25.0000, actualVelocity=-0.0300, debugActualVelocityA=-0.0300, debugActualVelocityB=-0.0300, odometer=392282.8600, timestamp=1667506455.3000
            1667506455.390: motors: raw=[-2453700000.0, -2451600000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
            1667506455.390: ccwFollowingError: positionError=-0.0000, velocityError=-0.0300, timestamp=1667506455.3000
            track angle=-25.35 velocity=-1.00
            1667506455.440: rotation: demandPosition=-25.0000, demandVelocity=-0.0300, demandAcceleration=-0.1200, actualPosition=-25.0000, actualVelocity=-0.0300, debugActualVelocityA=-0.0300, debugActualVelocityB=-0.0300, odometer=392282.8600, timestamp=1667506455.3500
            1667506455.490: motors: raw=[-2453600000.0, -2451500000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
            1667506455.490: target: position=-25.3514, velocity=-1.0000, tai=1667506455.4450
            1667506455.540: rotation: demandPosition=-25.0100, demandVelocity=-0.0400, demandAcceleration=-0.1200, actualPosition=-25.0100, actualVelocity=-0.0400, debugActualVelocityA=-0.0400, debugActualVelocityB=-0.0400, odometer=392282.8600, timestamp=1667506455.4500
            1667506455.540: motors: raw=[-2453600000.0, -2451400000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
            1667506455.540: ccwFollowingError: positionError=-0.0100, velocityError=-0.0400, timestamp=1667506455.4500
            1667506455.590: motors: raw=[-2453500000.0, -2451400000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
            track angle=-25.55 velocity=-1.00
            1667506455.640: rotation: demandPosition=-25.0100, demandVelocity=-0.0500, demandAcceleration=-0.1200, actualPosition=-25.0100, actualVelocity=-0.0500, debugActualVelocityA=-0.0500, debugActualVelocityB=-0.0500, odometer=392282.8600, timestamp=1667506455.5500
            1667506455.640: motors: raw=[-2453400000.0, -2451300000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
            1667506455.640: ccwFollowingError: positionError=-0.0100, velocityError=-0.0500, timestamp=1667506455.5500
            1667506455.690: rotation: demandPosition=-25.0100, demandVelocity=-0.0600, demandAcceleration=-0.1200, actualPosition=-25.0100, actualVelocity=-0.0600, debugActualVelocityA=-0.0600, debugActualVelocityB=-0.0600, odometer=392282.8700, timestamp=1667506455.6000
            1667506455.690: motors: raw=[-2453300000.0, -2451200000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
            1667506455.690: ccwFollowingError: positionError=-0.0100, velocityError=-0.0600, timestamp=1667506455.6000
            1667506455.690: target: position=-25.5512, velocity=-1.0000, tai=1667506455.6448
            1667506455.740: rotation: demandPosition=-25.0200, demandVelocity=-0.0600, demandAcceleration=-0.1200, actualPosition=-25.0200, actualVelocity=-0.0600, debugActualVelocityA=-0.0600, debugActualVelocityB=-0.0600, odometer=392282.8700, timestamp=1667506455.6500
            1667506455.740: motors: raw=[-2453200000.0, -2451100000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
            1667506455.741: ccwFollowingError: positionError=-0.0200, velocityError=-0.0600, timestamp=1667506455.6500
            1667506455.790: rotation: demandPosition=-25.0200, demandVelocity=-0.0700, demandAcceleration=-0.1200, actualPosition=-25.0200, actualVelocity=-0.0700, debugActualVelocityA=-0.0700, debugActualVelocityB=-0.0700, odometer=392282.8700, timestamp=1667506455.7000
            1667506455.790: motors: raw=[-2453100000.0, -2451000000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
            1667506455.791: ccwFollowingError: positionError=-0.0200, velocityError=-0.0700, timestamp=1667506455.7000
            track angle=-25.75 velocity=-1.00
            1667506455.840: rotation: demandPosition=-25.0200, demandVelocity=-0.0800, demandAcceleration=-0.1200, actualPosition=-25.0200, actualVelocity=-0.0800, debugActualVelocityA=-0.0800, debugActualVelocityB=-0.0800, odometer=392282.8800, timestamp=1667506455.7500
            1667506455.840: motors: raw=[-2452900000.0, -2450800000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
            1667506455.840: ccwFollowingError: positionError=-0.0200, velocityError=-0.0800, timestamp=1667506455.7500
            1667506455.890: rotation: demandPosition=-25.0300, demandVelocity=-0.0800, demandAcceleration=-0.1200, actualPosition=-25.0300, actualVelocity=-0.0800, debugActualVelocityA=-0.0800, debugActualVelocityB=-0.0800, odometer=392282.8800, timestamp=1667506455.8000
            1667506455.890: motors: raw=[-2452800000.0, -2450700000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
            1667506455.890: ccwFollowingError: positionError=-0.0300, velocityError=-0.0800, timestamp=1667506455.8000
            1667506455.890: target: position=-25.7517, velocity=-1.0000, tai=1667506455.8453
            1667506455.940: rotation: demandPosition=-25.0300, demandVelocity=-0.0900, demandAcceleration=-0.1200, actualPosition=-25.0300, actualVelocity=-0.0900, debugActualVelocityA=-0.0900, debugActualVelocityB=-0.0800, odometer=392282.8800, timestamp=1667506455.8500
            1667506455.940: motors: raw=[-2452600000.0, -2450500000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
            1667506455.940: ccwFollowingError: positionError=-0.0300, velocityError=-0.0900, timestamp=1667506455.8500
            1667506455.990: rotation: demandPosition=-25.0400, demandVelocity=-0.0900, demandAcceleration=-0.1200, actualPosition=-25.0400, actualVelocity=-0.0900, debugActualVelocityA=-0.0900, debugActualVelocityB=-0.0900, odometer=392282.8900, timestamp=1667506455.9000
            1667506455.990: motors: raw=[-2452500000.0, -2450300000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
            1667506455.990: ccwFollowingError: positionError=-0.0400, velocityError=-0.0900, timestamp=1667506455.9000
            track angle=-25.95 velocity=-1.00
            1667506456.040: rotation: demandPosition=-25.0400, demandVelocity=-0.1000, demandAcceleration=-0.1100, actualPosition=-25.0400, actualVelocity=-0.1000, debugActualVelocityA=-0.1000, debugActualVelocityB=-0.1000, odometer=392282.8900, timestamp=1667506455.9500
            1667506456.040: motors: raw=[-2452300000.0, -2450200000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
            1667506456.041: ccwFollowingError: positionError=-0.0400, velocityError=-0.1000, timestamp=1667506455.9500
            1667506456.090: rotation: demandPosition=-25.0500, demandVelocity=-0.1000, demandAcceleration=-0.0500, actualPosition=-25.0500, actualVelocity=-0.1000, debugActualVelocityA=-0.1000, debugActualVelocityB=-0.1100, odometer=392282.9000, timestamp=1667506456.0000
            1667506456.090: motors: raw=[-2452100000.0, -2450000000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
            1667506456.090: ccwFollowingError: positionError=-0.0500, velocityError=-0.1100, timestamp=1667506456.0000
            1667506456.091: target: position=-25.9516, velocity=-1.0000, tai=1667506456.0452
            1667506456.140: rotation: demandPosition=-25.0500, demandVelocity=-0.1000, demandAcceleration=0.0200, actualPosition=-25.0500, actualVelocity=-0.1000, debugActualVelocityA=-0.1100, debugActualVelocityB=-0.1000, odometer=392282.9000, timestamp=1667506456.0500
            1667506456.140: motors: raw=[-2451900000.0, -2449800000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
            1667506456.140: ccwFollowingError: positionError=-0.0500, velocityError=-0.1000, timestamp=1667506456.0500
            1667506456.190: rotation: demandPosition=-25.0600, demandVelocity=-0.1000, demandAcceleration=0.1600, actualPosition=-25.0600, actualVelocity=-0.1000, debugActualVelocityA=-0.1000, debugActualVelocityB=-0.1000, odometer=392282.9100, timestamp=1667506456.1000
            1667506456.190: motors: raw=[-2451700000.0, -2449600000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
            track angle=-26.00 velocity=0.00
            1667506456.240: rotation: demandPosition=-25.0600, demandVelocity=-0.0900, demandAcceleration=0.3000, actualPosition=-25.0600, actualVelocity=-0.0900, debugActualVelocityA=-0.0900, debugActualVelocityB=-0.1000, odometer=392282.9100, timestamp=1667506456.1500
            1667506456.240: motors: raw=[-2451500000.0, -2449400000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
            1667506456.240: ccwFollowingError: positionError=-0.0600, velocityError=-0.0900, timestamp=1667506456.1500
            1667506456.290: rotation: demandPosition=-25.0600, demandVelocity=-0.0700, demandAcceleration=0.3700, actualPosition=-25.0600, actualVelocity=-0.0700, debugActualVelocityA=-0.0700, debugActualVelocityB=-0.0700, odometer=392282.9200, timestamp=1667506456.2000
            1667506456.290: motors: raw=[-2451400000.0, -2449200000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
            1667506456.291: ccwFollowingError: positionError=-0.0600, velocityError=-0.0700, timestamp=1667506456.2000
            1667506456.290: tracking: tracking=False, lost=True, noNewCommand=False
            1667506456.291: target: position=-26.0000, velocity=0.0000, tai=1667506456.2452
            1667506456.290: inPosition: inPosition=False
            {code}

            This shows an attempt to track a path from -25 to -26 degrees at 1 deg/second (thus a very short ramp). The rotator starts moving in the correct direction but soon turns around. The tracking commands look like this: {{track angle=-25.95 velocity=-1.00}} (though you can also get the same information from the "target: " events, which the CSC publishes when it sends a track command or point-to-point move command to the low-level controller).

            I also attached a graph from [~julen.garcia] showing position vs time (of both rotator and camera cable wrap). This shows the rotator starting off in the correct direction, but then turning around and ending up in the wrong direction from the starting point. Also notice that the end of the motion is very abrupt; I doubt it is properly jerk-limited.
            rowen Russell Owen made changes -
            Link This issue relates to DM-36869 [ DM-36869 ]
            rowen Russell Owen made changes -
            Attachment failed ramp.txt [ 64440 ]
            rowen Russell Owen made changes -
            Link This issue relates to DM-36872 [ DM-36872 ]
            ttsai Te-Wei Tsai made changes -
            Link This issue relates to DM-25245 [ DM-25245 ]
            bquint Bruno Quint made changes -
            Attachment Rotator PosVelAcc.jpg [ 64469 ]
            bquint Bruno Quint made changes -
            Attachment Rotator PosVelAcc.jpg [ 64469 ]
            bquint Bruno Quint made changes -
            Attachment Rotator PosVelAcc.jpg [ 64470 ]
            rowen Russell Owen made changes -
            Watchers Brian Stalder, Bruno Quint, Russell Owen, Te-Wei Tsai [ Brian Stalder, Bruno Quint, Russell Owen, Te-Wei Tsai ] Brian Stalder, Bruno Quint, Julen Garcia, Russell Owen, Te-Wei Tsai [ Brian Stalder, Bruno Quint, Julen Garcia, Russell Owen, Te-Wei Tsai ]
            rowen Russell Owen made changes -
            Description In the last two days every time we try a ramp tracking sequence for the MT rotator from the MTRotator commander, the rotator moves in the wrong direction, often quite far in the wrong direction. Here is an example, with a very short ramp (so I didn't have to stop it):

            MT Rotator failed ramp 2022-11-03

            {code}
            ramp -25 -26 1
            run_command running command method <bound method RotatorCommander.do_ramp of <lsst.ts.mtrotator.rotator_commander.RotatorCommander object at 0x7fc1390205e0>>
            Tracking a ramp from -25.0 to -26.0 at speed 1.0; this will take 1.00 seconds
            Finished command ramp
            track angle=-25.00 velocity=-1.00
            1667506455.090: controllerState: controllerState=2, offlineSubstate=1, enabledSubstate=2, applicationStatus=5124
            1667506455.090: tracking: tracking=False, lost=True, noNewCommand=False
            1667506455.091: target: position=-25.0000, velocity=-1.0000, tai=1667506455.0936
            1667506455.090: inPosition: inPosition=False
            track angle=-25.15 velocity=-1.00
            1667506455.240: rotation: demandPosition=-25.0000, demandVelocity=-0.0000, demandAcceleration=-0.0600, actualPosition=-25.0000, actualVelocity=-0.0000, debugActualVelocityA=-0.0000, debugActualVelocityB=-0.0000, odometer=392282.8500, timestamp=1667506455.1500
            1667506455.240: tracking: tracking=True, lost=True, noNewCommand=False
            1667506455.240: inPosition: inPosition=True
            1667506455.290: rotation: demandPosition=-25.0000, demandVelocity=-0.0100, demandAcceleration=-0.1200, actualPosition=-25.0000, actualVelocity=-0.0000, debugActualVelocityA=-0.0000, debugActualVelocityB=-0.0000, odometer=392282.8500, timestamp=1667506455.2000
            1667506455.290: target: position=-25.1530, velocity=-1.0000, tai=1667506455.2466
            1667506455.340: rotation: demandPosition=-25.0000, demandVelocity=-0.0100, demandAcceleration=-0.1200, actualPosition=-25.0000, actualVelocity=-0.0200, debugActualVelocityA=-0.0200, debugActualVelocityB=-0.0100, odometer=392282.8500, timestamp=1667506455.2500
            1667506455.340: ccwFollowingError: positionError=-0.0000, velocityError=-0.0200, timestamp=1667506455.2500
            1667506455.390: rotation: demandPosition=-25.0000, demandVelocity=-0.0200, demandAcceleration=-0.1200, actualPosition=-25.0000, actualVelocity=-0.0300, debugActualVelocityA=-0.0300, debugActualVelocityB=-0.0300, odometer=392282.8600, timestamp=1667506455.3000
            1667506455.390: motors: raw=[-2453700000.0, -2451600000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
            1667506455.390: ccwFollowingError: positionError=-0.0000, velocityError=-0.0300, timestamp=1667506455.3000
            track angle=-25.35 velocity=-1.00
            1667506455.440: rotation: demandPosition=-25.0000, demandVelocity=-0.0300, demandAcceleration=-0.1200, actualPosition=-25.0000, actualVelocity=-0.0300, debugActualVelocityA=-0.0300, debugActualVelocityB=-0.0300, odometer=392282.8600, timestamp=1667506455.3500
            1667506455.490: motors: raw=[-2453600000.0, -2451500000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
            1667506455.490: target: position=-25.3514, velocity=-1.0000, tai=1667506455.4450
            1667506455.540: rotation: demandPosition=-25.0100, demandVelocity=-0.0400, demandAcceleration=-0.1200, actualPosition=-25.0100, actualVelocity=-0.0400, debugActualVelocityA=-0.0400, debugActualVelocityB=-0.0400, odometer=392282.8600, timestamp=1667506455.4500
            1667506455.540: motors: raw=[-2453600000.0, -2451400000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
            1667506455.540: ccwFollowingError: positionError=-0.0100, velocityError=-0.0400, timestamp=1667506455.4500
            1667506455.590: motors: raw=[-2453500000.0, -2451400000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
            track angle=-25.55 velocity=-1.00
            1667506455.640: rotation: demandPosition=-25.0100, demandVelocity=-0.0500, demandAcceleration=-0.1200, actualPosition=-25.0100, actualVelocity=-0.0500, debugActualVelocityA=-0.0500, debugActualVelocityB=-0.0500, odometer=392282.8600, timestamp=1667506455.5500
            1667506455.640: motors: raw=[-2453400000.0, -2451300000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
            1667506455.640: ccwFollowingError: positionError=-0.0100, velocityError=-0.0500, timestamp=1667506455.5500
            1667506455.690: rotation: demandPosition=-25.0100, demandVelocity=-0.0600, demandAcceleration=-0.1200, actualPosition=-25.0100, actualVelocity=-0.0600, debugActualVelocityA=-0.0600, debugActualVelocityB=-0.0600, odometer=392282.8700, timestamp=1667506455.6000
            1667506455.690: motors: raw=[-2453300000.0, -2451200000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
            1667506455.690: ccwFollowingError: positionError=-0.0100, velocityError=-0.0600, timestamp=1667506455.6000
            1667506455.690: target: position=-25.5512, velocity=-1.0000, tai=1667506455.6448
            1667506455.740: rotation: demandPosition=-25.0200, demandVelocity=-0.0600, demandAcceleration=-0.1200, actualPosition=-25.0200, actualVelocity=-0.0600, debugActualVelocityA=-0.0600, debugActualVelocityB=-0.0600, odometer=392282.8700, timestamp=1667506455.6500
            1667506455.740: motors: raw=[-2453200000.0, -2451100000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
            1667506455.741: ccwFollowingError: positionError=-0.0200, velocityError=-0.0600, timestamp=1667506455.6500
            1667506455.790: rotation: demandPosition=-25.0200, demandVelocity=-0.0700, demandAcceleration=-0.1200, actualPosition=-25.0200, actualVelocity=-0.0700, debugActualVelocityA=-0.0700, debugActualVelocityB=-0.0700, odometer=392282.8700, timestamp=1667506455.7000
            1667506455.790: motors: raw=[-2453100000.0, -2451000000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
            1667506455.791: ccwFollowingError: positionError=-0.0200, velocityError=-0.0700, timestamp=1667506455.7000
            track angle=-25.75 velocity=-1.00
            1667506455.840: rotation: demandPosition=-25.0200, demandVelocity=-0.0800, demandAcceleration=-0.1200, actualPosition=-25.0200, actualVelocity=-0.0800, debugActualVelocityA=-0.0800, debugActualVelocityB=-0.0800, odometer=392282.8800, timestamp=1667506455.7500
            1667506455.840: motors: raw=[-2452900000.0, -2450800000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
            1667506455.840: ccwFollowingError: positionError=-0.0200, velocityError=-0.0800, timestamp=1667506455.7500
            1667506455.890: rotation: demandPosition=-25.0300, demandVelocity=-0.0800, demandAcceleration=-0.1200, actualPosition=-25.0300, actualVelocity=-0.0800, debugActualVelocityA=-0.0800, debugActualVelocityB=-0.0800, odometer=392282.8800, timestamp=1667506455.8000
            1667506455.890: motors: raw=[-2452800000.0, -2450700000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
            1667506455.890: ccwFollowingError: positionError=-0.0300, velocityError=-0.0800, timestamp=1667506455.8000
            1667506455.890: target: position=-25.7517, velocity=-1.0000, tai=1667506455.8453
            1667506455.940: rotation: demandPosition=-25.0300, demandVelocity=-0.0900, demandAcceleration=-0.1200, actualPosition=-25.0300, actualVelocity=-0.0900, debugActualVelocityA=-0.0900, debugActualVelocityB=-0.0800, odometer=392282.8800, timestamp=1667506455.8500
            1667506455.940: motors: raw=[-2452600000.0, -2450500000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
            1667506455.940: ccwFollowingError: positionError=-0.0300, velocityError=-0.0900, timestamp=1667506455.8500
            1667506455.990: rotation: demandPosition=-25.0400, demandVelocity=-0.0900, demandAcceleration=-0.1200, actualPosition=-25.0400, actualVelocity=-0.0900, debugActualVelocityA=-0.0900, debugActualVelocityB=-0.0900, odometer=392282.8900, timestamp=1667506455.9000
            1667506455.990: motors: raw=[-2452500000.0, -2450300000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
            1667506455.990: ccwFollowingError: positionError=-0.0400, velocityError=-0.0900, timestamp=1667506455.9000
            track angle=-25.95 velocity=-1.00
            1667506456.040: rotation: demandPosition=-25.0400, demandVelocity=-0.1000, demandAcceleration=-0.1100, actualPosition=-25.0400, actualVelocity=-0.1000, debugActualVelocityA=-0.1000, debugActualVelocityB=-0.1000, odometer=392282.8900, timestamp=1667506455.9500
            1667506456.040: motors: raw=[-2452300000.0, -2450200000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
            1667506456.041: ccwFollowingError: positionError=-0.0400, velocityError=-0.1000, timestamp=1667506455.9500
            1667506456.090: rotation: demandPosition=-25.0500, demandVelocity=-0.1000, demandAcceleration=-0.0500, actualPosition=-25.0500, actualVelocity=-0.1000, debugActualVelocityA=-0.1000, debugActualVelocityB=-0.1100, odometer=392282.9000, timestamp=1667506456.0000
            1667506456.090: motors: raw=[-2452100000.0, -2450000000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
            1667506456.090: ccwFollowingError: positionError=-0.0500, velocityError=-0.1100, timestamp=1667506456.0000
            1667506456.091: target: position=-25.9516, velocity=-1.0000, tai=1667506456.0452
            1667506456.140: rotation: demandPosition=-25.0500, demandVelocity=-0.1000, demandAcceleration=0.0200, actualPosition=-25.0500, actualVelocity=-0.1000, debugActualVelocityA=-0.1100, debugActualVelocityB=-0.1000, odometer=392282.9000, timestamp=1667506456.0500
            1667506456.140: motors: raw=[-2451900000.0, -2449800000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
            1667506456.140: ccwFollowingError: positionError=-0.0500, velocityError=-0.1000, timestamp=1667506456.0500
            1667506456.190: rotation: demandPosition=-25.0600, demandVelocity=-0.1000, demandAcceleration=0.1600, actualPosition=-25.0600, actualVelocity=-0.1000, debugActualVelocityA=-0.1000, debugActualVelocityB=-0.1000, odometer=392282.9100, timestamp=1667506456.1000
            1667506456.190: motors: raw=[-2451700000.0, -2449600000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
            track angle=-26.00 velocity=0.00
            1667506456.240: rotation: demandPosition=-25.0600, demandVelocity=-0.0900, demandAcceleration=0.3000, actualPosition=-25.0600, actualVelocity=-0.0900, debugActualVelocityA=-0.0900, debugActualVelocityB=-0.1000, odometer=392282.9100, timestamp=1667506456.1500
            1667506456.240: motors: raw=[-2451500000.0, -2449400000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
            1667506456.240: ccwFollowingError: positionError=-0.0600, velocityError=-0.0900, timestamp=1667506456.1500
            1667506456.290: rotation: demandPosition=-25.0600, demandVelocity=-0.0700, demandAcceleration=0.3700, actualPosition=-25.0600, actualVelocity=-0.0700, debugActualVelocityA=-0.0700, debugActualVelocityB=-0.0700, odometer=392282.9200, timestamp=1667506456.2000
            1667506456.290: motors: raw=[-2451400000.0, -2449200000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
            1667506456.291: ccwFollowingError: positionError=-0.0600, velocityError=-0.0700, timestamp=1667506456.2000
            1667506456.290: tracking: tracking=False, lost=True, noNewCommand=False
            1667506456.291: target: position=-26.0000, velocity=0.0000, tai=1667506456.2452
            1667506456.290: inPosition: inPosition=False
            {code}

            This shows an attempt to track a path from -25 to -26 degrees at 1 deg/second (thus a very short ramp). The rotator starts moving in the correct direction but soon turns around. The tracking commands look like this: {{track angle=-25.95 velocity=-1.00}} (though you can also get the same information from the "target: " events, which the CSC publishes when it sends a track command or point-to-point move command to the low-level controller).

            I also attached a graph from [~julen.garcia] showing position vs time (of both rotator and camera cable wrap). This shows the rotator starting off in the correct direction, but then turning around and ending up in the wrong direction from the starting point. Also notice that the end of the motion is very abrupt; I doubt it is properly jerk-limited.             		In the last two days every time we try a ramp tracking sequence for the MT rotator from the MTRotator commander, the rotator moves in the wrong direction, often quite far in the wrong direction. Here is an example, with a very short ramp (so I didn't have to stop it), but the examples attached are much easier to understand. See the graph

            MT Rotator failed ramp 2022-11-03

            {code}
            ramp -25 -26 1
            run_command running command method <bound method RotatorCommander.do_ramp of <lsst.ts.mtrotator.rotator_commander.RotatorCommander object at 0x7fc1390205e0>>
            Tracking a ramp from -25.0 to -26.0 at speed 1.0; this will take 1.00 seconds
            Finished command ramp
            track angle=-25.00 velocity=-1.00
            1667506455.090: controllerState: controllerState=2, offlineSubstate=1, enabledSubstate=2, applicationStatus=5124
            1667506455.090: tracking: tracking=False, lost=True, noNewCommand=False
            1667506455.091: target: position=-25.0000, velocity=-1.0000, tai=1667506455.0936
            1667506455.090: inPosition: inPosition=False
            track angle=-25.15 velocity=-1.00
            1667506455.240: rotation: demandPosition=-25.0000, demandVelocity=-0.0000, demandAcceleration=-0.0600, actualPosition=-25.0000, actualVelocity=-0.0000, debugActualVelocityA=-0.0000, debugActualVelocityB=-0.0000, odometer=392282.8500, timestamp=1667506455.1500
            1667506455.240: tracking: tracking=True, lost=True, noNewCommand=False
            1667506455.240: inPosition: inPosition=True
            1667506455.290: rotation: demandPosition=-25.0000, demandVelocity=-0.0100, demandAcceleration=-0.1200, actualPosition=-25.0000, actualVelocity=-0.0000, debugActualVelocityA=-0.0000, debugActualVelocityB=-0.0000, odometer=392282.8500, timestamp=1667506455.2000
            1667506455.290: target: position=-25.1530, velocity=-1.0000, tai=1667506455.2466
            1667506455.340: rotation: demandPosition=-25.0000, demandVelocity=-0.0100, demandAcceleration=-0.1200, actualPosition=-25.0000, actualVelocity=-0.0200, debugActualVelocityA=-0.0200, debugActualVelocityB=-0.0100, odometer=392282.8500, timestamp=1667506455.2500
            1667506455.340: ccwFollowingError: positionError=-0.0000, velocityError=-0.0200, timestamp=1667506455.2500
            1667506455.390: rotation: demandPosition=-25.0000, demandVelocity=-0.0200, demandAcceleration=-0.1200, actualPosition=-25.0000, actualVelocity=-0.0300, debugActualVelocityA=-0.0300, debugActualVelocityB=-0.0300, odometer=392282.8600, timestamp=1667506455.3000
            1667506455.390: motors: raw=[-2453700000.0, -2451600000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
            1667506455.390: ccwFollowingError: positionError=-0.0000, velocityError=-0.0300, timestamp=1667506455.3000
            track angle=-25.35 velocity=-1.00
            1667506455.440: rotation: demandPosition=-25.0000, demandVelocity=-0.0300, demandAcceleration=-0.1200, actualPosition=-25.0000, actualVelocity=-0.0300, debugActualVelocityA=-0.0300, debugActualVelocityB=-0.0300, odometer=392282.8600, timestamp=1667506455.3500
            1667506455.490: motors: raw=[-2453600000.0, -2451500000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
            1667506455.490: target: position=-25.3514, velocity=-1.0000, tai=1667506455.4450
            1667506455.540: rotation: demandPosition=-25.0100, demandVelocity=-0.0400, demandAcceleration=-0.1200, actualPosition=-25.0100, actualVelocity=-0.0400, debugActualVelocityA=-0.0400, debugActualVelocityB=-0.0400, odometer=392282.8600, timestamp=1667506455.4500
            1667506455.540: motors: raw=[-2453600000.0, -2451400000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
            1667506455.540: ccwFollowingError: positionError=-0.0100, velocityError=-0.0400, timestamp=1667506455.4500
            1667506455.590: motors: raw=[-2453500000.0, -2451400000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
            track angle=-25.55 velocity=-1.00
            1667506455.640: rotation: demandPosition=-25.0100, demandVelocity=-0.0500, demandAcceleration=-0.1200, actualPosition=-25.0100, actualVelocity=-0.0500, debugActualVelocityA=-0.0500, debugActualVelocityB=-0.0500, odometer=392282.8600, timestamp=1667506455.5500
            1667506455.640: motors: raw=[-2453400000.0, -2451300000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
            1667506455.640: ccwFollowingError: positionError=-0.0100, velocityError=-0.0500, timestamp=1667506455.5500
            1667506455.690: rotation: demandPosition=-25.0100, demandVelocity=-0.0600, demandAcceleration=-0.1200, actualPosition=-25.0100, actualVelocity=-0.0600, debugActualVelocityA=-0.0600, debugActualVelocityB=-0.0600, odometer=392282.8700, timestamp=1667506455.6000
            1667506455.690: motors: raw=[-2453300000.0, -2451200000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
            1667506455.690: ccwFollowingError: positionError=-0.0100, velocityError=-0.0600, timestamp=1667506455.6000
            1667506455.690: target: position=-25.5512, velocity=-1.0000, tai=1667506455.6448
            1667506455.740: rotation: demandPosition=-25.0200, demandVelocity=-0.0600, demandAcceleration=-0.1200, actualPosition=-25.0200, actualVelocity=-0.0600, debugActualVelocityA=-0.0600, debugActualVelocityB=-0.0600, odometer=392282.8700, timestamp=1667506455.6500
            1667506455.740: motors: raw=[-2453200000.0, -2451100000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
            1667506455.741: ccwFollowingError: positionError=-0.0200, velocityError=-0.0600, timestamp=1667506455.6500
            1667506455.790: rotation: demandPosition=-25.0200, demandVelocity=-0.0700, demandAcceleration=-0.1200, actualPosition=-25.0200, actualVelocity=-0.0700, debugActualVelocityA=-0.0700, debugActualVelocityB=-0.0700, odometer=392282.8700, timestamp=1667506455.7000
            1667506455.790: motors: raw=[-2453100000.0, -2451000000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
            1667506455.791: ccwFollowingError: positionError=-0.0200, velocityError=-0.0700, timestamp=1667506455.7000
            track angle=-25.75 velocity=-1.00
            1667506455.840: rotation: demandPosition=-25.0200, demandVelocity=-0.0800, demandAcceleration=-0.1200, actualPosition=-25.0200, actualVelocity=-0.0800, debugActualVelocityA=-0.0800, debugActualVelocityB=-0.0800, odometer=392282.8800, timestamp=1667506455.7500
            1667506455.840: motors: raw=[-2452900000.0, -2450800000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
            1667506455.840: ccwFollowingError: positionError=-0.0200, velocityError=-0.0800, timestamp=1667506455.7500
            1667506455.890: rotation: demandPosition=-25.0300, demandVelocity=-0.0800, demandAcceleration=-0.1200, actualPosition=-25.0300, actualVelocity=-0.0800, debugActualVelocityA=-0.0800, debugActualVelocityB=-0.0800, odometer=392282.8800, timestamp=1667506455.8000
            1667506455.890: motors: raw=[-2452800000.0, -2450700000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
            1667506455.890: ccwFollowingError: positionError=-0.0300, velocityError=-0.0800, timestamp=1667506455.8000
            1667506455.890: target: position=-25.7517, velocity=-1.0000, tai=1667506455.8453
            1667506455.940: rotation: demandPosition=-25.0300, demandVelocity=-0.0900, demandAcceleration=-0.1200, actualPosition=-25.0300, actualVelocity=-0.0900, debugActualVelocityA=-0.0900, debugActualVelocityB=-0.0800, odometer=392282.8800, timestamp=1667506455.8500
            1667506455.940: motors: raw=[-2452600000.0, -2450500000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
            1667506455.940: ccwFollowingError: positionError=-0.0300, velocityError=-0.0900, timestamp=1667506455.8500
            1667506455.990: rotation: demandPosition=-25.0400, demandVelocity=-0.0900, demandAcceleration=-0.1200, actualPosition=-25.0400, actualVelocity=-0.0900, debugActualVelocityA=-0.0900, debugActualVelocityB=-0.0900, odometer=392282.8900, timestamp=1667506455.9000
            1667506455.990: motors: raw=[-2452500000.0, -2450300000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
            1667506455.990: ccwFollowingError: positionError=-0.0400, velocityError=-0.0900, timestamp=1667506455.9000
            track angle=-25.95 velocity=-1.00
            1667506456.040: rotation: demandPosition=-25.0400, demandVelocity=-0.1000, demandAcceleration=-0.1100, actualPosition=-25.0400, actualVelocity=-0.1000, debugActualVelocityA=-0.1000, debugActualVelocityB=-0.1000, odometer=392282.8900, timestamp=1667506455.9500
            1667506456.040: motors: raw=[-2452300000.0, -2450200000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
            1667506456.041: ccwFollowingError: positionError=-0.0400, velocityError=-0.1000, timestamp=1667506455.9500
            1667506456.090: rotation: demandPosition=-25.0500, demandVelocity=-0.1000, demandAcceleration=-0.0500, actualPosition=-25.0500, actualVelocity=-0.1000, debugActualVelocityA=-0.1000, debugActualVelocityB=-0.1100, odometer=392282.9000, timestamp=1667506456.0000
            1667506456.090: motors: raw=[-2452100000.0, -2450000000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
            1667506456.090: ccwFollowingError: positionError=-0.0500, velocityError=-0.1100, timestamp=1667506456.0000
            1667506456.091: target: position=-25.9516, velocity=-1.0000, tai=1667506456.0452
            1667506456.140: rotation: demandPosition=-25.0500, demandVelocity=-0.1000, demandAcceleration=0.0200, actualPosition=-25.0500, actualVelocity=-0.1000, debugActualVelocityA=-0.1100, debugActualVelocityB=-0.1000, odometer=392282.9000, timestamp=1667506456.0500
            1667506456.140: motors: raw=[-2451900000.0, -2449800000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
            1667506456.140: ccwFollowingError: positionError=-0.0500, velocityError=-0.1000, timestamp=1667506456.0500
            1667506456.190: rotation: demandPosition=-25.0600, demandVelocity=-0.1000, demandAcceleration=0.1600, actualPosition=-25.0600, actualVelocity=-0.1000, debugActualVelocityA=-0.1000, debugActualVelocityB=-0.1000, odometer=392282.9100, timestamp=1667506456.1000
            1667506456.190: motors: raw=[-2451700000.0, -2449600000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
            track angle=-26.00 velocity=0.00
            1667506456.240: rotation: demandPosition=-25.0600, demandVelocity=-0.0900, demandAcceleration=0.3000, actualPosition=-25.0600, actualVelocity=-0.0900, debugActualVelocityA=-0.0900, debugActualVelocityB=-0.1000, odometer=392282.9100, timestamp=1667506456.1500
            1667506456.240: motors: raw=[-2451500000.0, -2449400000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
            1667506456.240: ccwFollowingError: positionError=-0.0600, velocityError=-0.0900, timestamp=1667506456.1500
            1667506456.290: rotation: demandPosition=-25.0600, demandVelocity=-0.0700, demandAcceleration=0.3700, actualPosition=-25.0600, actualVelocity=-0.0700, debugActualVelocityA=-0.0700, debugActualVelocityB=-0.0700, odometer=392282.9200, timestamp=1667506456.2000
            1667506456.290: motors: raw=[-2451400000.0, -2449200000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
            1667506456.291: ccwFollowingError: positionError=-0.0600, velocityError=-0.0700, timestamp=1667506456.2000
            1667506456.290: tracking: tracking=False, lost=True, noNewCommand=False
            1667506456.291: target: position=-26.0000, velocity=0.0000, tai=1667506456.2452
            1667506456.290: inPosition: inPosition=False
            {code}

            This shows an attempt to track a path from -25 to -26 degrees at 1 deg/second (thus a very short ramp). The rotator starts moving in the correct direction but soon turns around. The tracking commands look like this: {{track angle=-25.95 velocity=-1.00}} (though you can also get the same information from the "target: " events, which the CSC publishes when it sends a track command or point-to-point move command to the low-level controller).

            I also attached a graph from [~julen.garcia] showing position vs time (of both rotator and camera cable wrap). This shows the rotator starting off in the correct direction, but then turning around and ending up in the wrong direction from the starting point. Also notice that the end of the motion is very abrupt; I doubt it is properly jerk-limited.
            rowen Russell Owen made changes -
            Description In the last two days every time we try a ramp tracking sequence for the MT rotator from the MTRotator commander, the rotator moves in the wrong direction, often quite far in the wrong direction. Here is an example, with a very short ramp (so I didn't have to stop it), but the examples attached are much easier to understand. See the graph

            MT Rotator failed ramp 2022-11-03

            {code}
            ramp -25 -26 1
            run_command running command method <bound method RotatorCommander.do_ramp of <lsst.ts.mtrotator.rotator_commander.RotatorCommander object at 0x7fc1390205e0>>
            Tracking a ramp from -25.0 to -26.0 at speed 1.0; this will take 1.00 seconds
            Finished command ramp
            track angle=-25.00 velocity=-1.00
            1667506455.090: controllerState: controllerState=2, offlineSubstate=1, enabledSubstate=2, applicationStatus=5124
            1667506455.090: tracking: tracking=False, lost=True, noNewCommand=False
            1667506455.091: target: position=-25.0000, velocity=-1.0000, tai=1667506455.0936
            1667506455.090: inPosition: inPosition=False
            track angle=-25.15 velocity=-1.00
            1667506455.240: rotation: demandPosition=-25.0000, demandVelocity=-0.0000, demandAcceleration=-0.0600, actualPosition=-25.0000, actualVelocity=-0.0000, debugActualVelocityA=-0.0000, debugActualVelocityB=-0.0000, odometer=392282.8500, timestamp=1667506455.1500
            1667506455.240: tracking: tracking=True, lost=True, noNewCommand=False
            1667506455.240: inPosition: inPosition=True
            1667506455.290: rotation: demandPosition=-25.0000, demandVelocity=-0.0100, demandAcceleration=-0.1200, actualPosition=-25.0000, actualVelocity=-0.0000, debugActualVelocityA=-0.0000, debugActualVelocityB=-0.0000, odometer=392282.8500, timestamp=1667506455.2000
            1667506455.290: target: position=-25.1530, velocity=-1.0000, tai=1667506455.2466
            1667506455.340: rotation: demandPosition=-25.0000, demandVelocity=-0.0100, demandAcceleration=-0.1200, actualPosition=-25.0000, actualVelocity=-0.0200, debugActualVelocityA=-0.0200, debugActualVelocityB=-0.0100, odometer=392282.8500, timestamp=1667506455.2500
            1667506455.340: ccwFollowingError: positionError=-0.0000, velocityError=-0.0200, timestamp=1667506455.2500
            1667506455.390: rotation: demandPosition=-25.0000, demandVelocity=-0.0200, demandAcceleration=-0.1200, actualPosition=-25.0000, actualVelocity=-0.0300, debugActualVelocityA=-0.0300, debugActualVelocityB=-0.0300, odometer=392282.8600, timestamp=1667506455.3000
            1667506455.390: motors: raw=[-2453700000.0, -2451600000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
            1667506455.390: ccwFollowingError: positionError=-0.0000, velocityError=-0.0300, timestamp=1667506455.3000
            track angle=-25.35 velocity=-1.00
            1667506455.440: rotation: demandPosition=-25.0000, demandVelocity=-0.0300, demandAcceleration=-0.1200, actualPosition=-25.0000, actualVelocity=-0.0300, debugActualVelocityA=-0.0300, debugActualVelocityB=-0.0300, odometer=392282.8600, timestamp=1667506455.3500
            1667506455.490: motors: raw=[-2453600000.0, -2451500000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
            1667506455.490: target: position=-25.3514, velocity=-1.0000, tai=1667506455.4450
            1667506455.540: rotation: demandPosition=-25.0100, demandVelocity=-0.0400, demandAcceleration=-0.1200, actualPosition=-25.0100, actualVelocity=-0.0400, debugActualVelocityA=-0.0400, debugActualVelocityB=-0.0400, odometer=392282.8600, timestamp=1667506455.4500
            1667506455.540: motors: raw=[-2453600000.0, -2451400000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
            1667506455.540: ccwFollowingError: positionError=-0.0100, velocityError=-0.0400, timestamp=1667506455.4500
            1667506455.590: motors: raw=[-2453500000.0, -2451400000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
            track angle=-25.55 velocity=-1.00
            1667506455.640: rotation: demandPosition=-25.0100, demandVelocity=-0.0500, demandAcceleration=-0.1200, actualPosition=-25.0100, actualVelocity=-0.0500, debugActualVelocityA=-0.0500, debugActualVelocityB=-0.0500, odometer=392282.8600, timestamp=1667506455.5500
            1667506455.640: motors: raw=[-2453400000.0, -2451300000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
            1667506455.640: ccwFollowingError: positionError=-0.0100, velocityError=-0.0500, timestamp=1667506455.5500
            1667506455.690: rotation: demandPosition=-25.0100, demandVelocity=-0.0600, demandAcceleration=-0.1200, actualPosition=-25.0100, actualVelocity=-0.0600, debugActualVelocityA=-0.0600, debugActualVelocityB=-0.0600, odometer=392282.8700, timestamp=1667506455.6000
            1667506455.690: motors: raw=[-2453300000.0, -2451200000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
            1667506455.690: ccwFollowingError: positionError=-0.0100, velocityError=-0.0600, timestamp=1667506455.6000
            1667506455.690: target: position=-25.5512, velocity=-1.0000, tai=1667506455.6448
            1667506455.740: rotation: demandPosition=-25.0200, demandVelocity=-0.0600, demandAcceleration=-0.1200, actualPosition=-25.0200, actualVelocity=-0.0600, debugActualVelocityA=-0.0600, debugActualVelocityB=-0.0600, odometer=392282.8700, timestamp=1667506455.6500
            1667506455.740: motors: raw=[-2453200000.0, -2451100000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
            1667506455.741: ccwFollowingError: positionError=-0.0200, velocityError=-0.0600, timestamp=1667506455.6500
            1667506455.790: rotation: demandPosition=-25.0200, demandVelocity=-0.0700, demandAcceleration=-0.1200, actualPosition=-25.0200, actualVelocity=-0.0700, debugActualVelocityA=-0.0700, debugActualVelocityB=-0.0700, odometer=392282.8700, timestamp=1667506455.7000
            1667506455.790: motors: raw=[-2453100000.0, -2451000000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
            1667506455.791: ccwFollowingError: positionError=-0.0200, velocityError=-0.0700, timestamp=1667506455.7000
            track angle=-25.75 velocity=-1.00
            1667506455.840: rotation: demandPosition=-25.0200, demandVelocity=-0.0800, demandAcceleration=-0.1200, actualPosition=-25.0200, actualVelocity=-0.0800, debugActualVelocityA=-0.0800, debugActualVelocityB=-0.0800, odometer=392282.8800, timestamp=1667506455.7500
            1667506455.840: motors: raw=[-2452900000.0, -2450800000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
            1667506455.840: ccwFollowingError: positionError=-0.0200, velocityError=-0.0800, timestamp=1667506455.7500
            1667506455.890: rotation: demandPosition=-25.0300, demandVelocity=-0.0800, demandAcceleration=-0.1200, actualPosition=-25.0300, actualVelocity=-0.0800, debugActualVelocityA=-0.0800, debugActualVelocityB=-0.0800, odometer=392282.8800, timestamp=1667506455.8000
            1667506455.890: motors: raw=[-2452800000.0, -2450700000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
            1667506455.890: ccwFollowingError: positionError=-0.0300, velocityError=-0.0800, timestamp=1667506455.8000
            1667506455.890: target: position=-25.7517, velocity=-1.0000, tai=1667506455.8453
            1667506455.940: rotation: demandPosition=-25.0300, demandVelocity=-0.0900, demandAcceleration=-0.1200, actualPosition=-25.0300, actualVelocity=-0.0900, debugActualVelocityA=-0.0900, debugActualVelocityB=-0.0800, odometer=392282.8800, timestamp=1667506455.8500
            1667506455.940: motors: raw=[-2452600000.0, -2450500000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
            1667506455.940: ccwFollowingError: positionError=-0.0300, velocityError=-0.0900, timestamp=1667506455.8500
            1667506455.990: rotation: demandPosition=-25.0400, demandVelocity=-0.0900, demandAcceleration=-0.1200, actualPosition=-25.0400, actualVelocity=-0.0900, debugActualVelocityA=-0.0900, debugActualVelocityB=-0.0900, odometer=392282.8900, timestamp=1667506455.9000
            1667506455.990: motors: raw=[-2452500000.0, -2450300000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
            1667506455.990: ccwFollowingError: positionError=-0.0400, velocityError=-0.0900, timestamp=1667506455.9000
            track angle=-25.95 velocity=-1.00
            1667506456.040: rotation: demandPosition=-25.0400, demandVelocity=-0.1000, demandAcceleration=-0.1100, actualPosition=-25.0400, actualVelocity=-0.1000, debugActualVelocityA=-0.1000, debugActualVelocityB=-0.1000, odometer=392282.8900, timestamp=1667506455.9500
            1667506456.040: motors: raw=[-2452300000.0, -2450200000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
            1667506456.041: ccwFollowingError: positionError=-0.0400, velocityError=-0.1000, timestamp=1667506455.9500
            1667506456.090: rotation: demandPosition=-25.0500, demandVelocity=-0.1000, demandAcceleration=-0.0500, actualPosition=-25.0500, actualVelocity=-0.1000, debugActualVelocityA=-0.1000, debugActualVelocityB=-0.1100, odometer=392282.9000, timestamp=1667506456.0000
            1667506456.090: motors: raw=[-2452100000.0, -2450000000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
            1667506456.090: ccwFollowingError: positionError=-0.0500, velocityError=-0.1100, timestamp=1667506456.0000
            1667506456.091: target: position=-25.9516, velocity=-1.0000, tai=1667506456.0452
            1667506456.140: rotation: demandPosition=-25.0500, demandVelocity=-0.1000, demandAcceleration=0.0200, actualPosition=-25.0500, actualVelocity=-0.1000, debugActualVelocityA=-0.1100, debugActualVelocityB=-0.1000, odometer=392282.9000, timestamp=1667506456.0500
            1667506456.140: motors: raw=[-2451900000.0, -2449800000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
            1667506456.140: ccwFollowingError: positionError=-0.0500, velocityError=-0.1000, timestamp=1667506456.0500
            1667506456.190: rotation: demandPosition=-25.0600, demandVelocity=-0.1000, demandAcceleration=0.1600, actualPosition=-25.0600, actualVelocity=-0.1000, debugActualVelocityA=-0.1000, debugActualVelocityB=-0.1000, odometer=392282.9100, timestamp=1667506456.1000
            1667506456.190: motors: raw=[-2451700000.0, -2449600000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
            track angle=-26.00 velocity=0.00
            1667506456.240: rotation: demandPosition=-25.0600, demandVelocity=-0.0900, demandAcceleration=0.3000, actualPosition=-25.0600, actualVelocity=-0.0900, debugActualVelocityA=-0.0900, debugActualVelocityB=-0.1000, odometer=392282.9100, timestamp=1667506456.1500
            1667506456.240: motors: raw=[-2451500000.0, -2449400000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
            1667506456.240: ccwFollowingError: positionError=-0.0600, velocityError=-0.0900, timestamp=1667506456.1500
            1667506456.290: rotation: demandPosition=-25.0600, demandVelocity=-0.0700, demandAcceleration=0.3700, actualPosition=-25.0600, actualVelocity=-0.0700, debugActualVelocityA=-0.0700, debugActualVelocityB=-0.0700, odometer=392282.9200, timestamp=1667506456.2000
            1667506456.290: motors: raw=[-2451400000.0, -2449200000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
            1667506456.291: ccwFollowingError: positionError=-0.0600, velocityError=-0.0700, timestamp=1667506456.2000
            1667506456.290: tracking: tracking=False, lost=True, noNewCommand=False
            1667506456.291: target: position=-26.0000, velocity=0.0000, tai=1667506456.2452
            1667506456.290: inPosition: inPosition=False
            {code}

            This shows an attempt to track a path from -25 to -26 degrees at 1 deg/second (thus a very short ramp). The rotator starts moving in the correct direction but soon turns around. The tracking commands look like this: {{track angle=-25.95 velocity=-1.00}} (though you can also get the same information from the "target: " events, which the CSC publishes when it sends a track command or point-to-point move command to the low-level controller).

            I also attached a graph from [~julen.garcia] showing position vs time (of both rotator and camera cable wrap). This shows the rotator starting off in the correct direction, but then turning around and ending up in the wrong direction from the starting point. Also notice that the end of the motion is very abrupt; I doubt it is properly jerk-limited.             		In the last two days every time we try a ramp tracking sequence for the MT rotator from the MTRotator commander, the rotator moves in the wrong direction, often quite far in the wrong direction. Here is an example, with a very short ramp (so I didn't have to stop it):

            MT Rotator failed ramp 2022-11-03

            {code}
            ramp -25 -26 1
            run_command running command method <bound method RotatorCommander.do_ramp of <lsst.ts.mtrotator.rotator_commander.RotatorCommander object at 0x7fc1390205e0>>
            Tracking a ramp from -25.0 to -26.0 at speed 1.0; this will take 1.00 seconds
            Finished command ramp
            track angle=-25.00 velocity=-1.00
            1667506455.090: controllerState: controllerState=2, offlineSubstate=1, enabledSubstate=2, applicationStatus=5124
            1667506455.090: tracking: tracking=False, lost=True, noNewCommand=False
            1667506455.091: target: position=-25.0000, velocity=-1.0000, tai=1667506455.0936
            1667506455.090: inPosition: inPosition=False
            track angle=-25.15 velocity=-1.00
            1667506455.240: rotation: demandPosition=-25.0000, demandVelocity=-0.0000, demandAcceleration=-0.0600, actualPosition=-25.0000, actualVelocity=-0.0000, debugActualVelocityA=-0.0000, debugActualVelocityB=-0.0000, odometer=392282.8500, timestamp=1667506455.1500
            1667506455.240: tracking: tracking=True, lost=True, noNewCommand=False
            1667506455.240: inPosition: inPosition=True
            1667506455.290: rotation: demandPosition=-25.0000, demandVelocity=-0.0100, demandAcceleration=-0.1200, actualPosition=-25.0000, actualVelocity=-0.0000, debugActualVelocityA=-0.0000, debugActualVelocityB=-0.0000, odometer=392282.8500, timestamp=1667506455.2000
            1667506455.290: target: position=-25.1530, velocity=-1.0000, tai=1667506455.2466
            1667506455.340: rotation: demandPosition=-25.0000, demandVelocity=-0.0100, demandAcceleration=-0.1200, actualPosition=-25.0000, actualVelocity=-0.0200, debugActualVelocityA=-0.0200, debugActualVelocityB=-0.0100, odometer=392282.8500, timestamp=1667506455.2500
            1667506455.340: ccwFollowingError: positionError=-0.0000, velocityError=-0.0200, timestamp=1667506455.2500
            1667506455.390: rotation: demandPosition=-25.0000, demandVelocity=-0.0200, demandAcceleration=-0.1200, actualPosition=-25.0000, actualVelocity=-0.0300, debugActualVelocityA=-0.0300, debugActualVelocityB=-0.0300, odometer=392282.8600, timestamp=1667506455.3000
            1667506455.390: motors: raw=[-2453700000.0, -2451600000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
            1667506455.390: ccwFollowingError: positionError=-0.0000, velocityError=-0.0300, timestamp=1667506455.3000
            track angle=-25.35 velocity=-1.00
            1667506455.440: rotation: demandPosition=-25.0000, demandVelocity=-0.0300, demandAcceleration=-0.1200, actualPosition=-25.0000, actualVelocity=-0.0300, debugActualVelocityA=-0.0300, debugActualVelocityB=-0.0300, odometer=392282.8600, timestamp=1667506455.3500
            1667506455.490: motors: raw=[-2453600000.0, -2451500000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
            1667506455.490: target: position=-25.3514, velocity=-1.0000, tai=1667506455.4450
            1667506455.540: rotation: demandPosition=-25.0100, demandVelocity=-0.0400, demandAcceleration=-0.1200, actualPosition=-25.0100, actualVelocity=-0.0400, debugActualVelocityA=-0.0400, debugActualVelocityB=-0.0400, odometer=392282.8600, timestamp=1667506455.4500
            1667506455.540: motors: raw=[-2453600000.0, -2451400000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
            1667506455.540: ccwFollowingError: positionError=-0.0100, velocityError=-0.0400, timestamp=1667506455.4500
            1667506455.590: motors: raw=[-2453500000.0, -2451400000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
            track angle=-25.55 velocity=-1.00
            1667506455.640: rotation: demandPosition=-25.0100, demandVelocity=-0.0500, demandAcceleration=-0.1200, actualPosition=-25.0100, actualVelocity=-0.0500, debugActualVelocityA=-0.0500, debugActualVelocityB=-0.0500, odometer=392282.8600, timestamp=1667506455.5500
            1667506455.640: motors: raw=[-2453400000.0, -2451300000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
            1667506455.640: ccwFollowingError: positionError=-0.0100, velocityError=-0.0500, timestamp=1667506455.5500
            1667506455.690: rotation: demandPosition=-25.0100, demandVelocity=-0.0600, demandAcceleration=-0.1200, actualPosition=-25.0100, actualVelocity=-0.0600, debugActualVelocityA=-0.0600, debugActualVelocityB=-0.0600, odometer=392282.8700, timestamp=1667506455.6000
            1667506455.690: motors: raw=[-2453300000.0, -2451200000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
            1667506455.690: ccwFollowingError: positionError=-0.0100, velocityError=-0.0600, timestamp=1667506455.6000
            1667506455.690: target: position=-25.5512, velocity=-1.0000, tai=1667506455.6448
            1667506455.740: rotation: demandPosition=-25.0200, demandVelocity=-0.0600, demandAcceleration=-0.1200, actualPosition=-25.0200, actualVelocity=-0.0600, debugActualVelocityA=-0.0600, debugActualVelocityB=-0.0600, odometer=392282.8700, timestamp=1667506455.6500
            1667506455.740: motors: raw=[-2453200000.0, -2451100000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
            1667506455.741: ccwFollowingError: positionError=-0.0200, velocityError=-0.0600, timestamp=1667506455.6500
            1667506455.790: rotation: demandPosition=-25.0200, demandVelocity=-0.0700, demandAcceleration=-0.1200, actualPosition=-25.0200, actualVelocity=-0.0700, debugActualVelocityA=-0.0700, debugActualVelocityB=-0.0700, odometer=392282.8700, timestamp=1667506455.7000
            1667506455.790: motors: raw=[-2453100000.0, -2451000000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
            1667506455.791: ccwFollowingError: positionError=-0.0200, velocityError=-0.0700, timestamp=1667506455.7000
            track angle=-25.75 velocity=-1.00
            1667506455.840: rotation: demandPosition=-25.0200, demandVelocity=-0.0800, demandAcceleration=-0.1200, actualPosition=-25.0200, actualVelocity=-0.0800, debugActualVelocityA=-0.0800, debugActualVelocityB=-0.0800, odometer=392282.8800, timestamp=1667506455.7500
            1667506455.840: motors: raw=[-2452900000.0, -2450800000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
            1667506455.840: ccwFollowingError: positionError=-0.0200, velocityError=-0.0800, timestamp=1667506455.7500
            1667506455.890: rotation: demandPosition=-25.0300, demandVelocity=-0.0800, demandAcceleration=-0.1200, actualPosition=-25.0300, actualVelocity=-0.0800, debugActualVelocityA=-0.0800, debugActualVelocityB=-0.0800, odometer=392282.8800, timestamp=1667506455.8000
            1667506455.890: motors: raw=[-2452800000.0, -2450700000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
            1667506455.890: ccwFollowingError: positionError=-0.0300, velocityError=-0.0800, timestamp=1667506455.8000
            1667506455.890: target: position=-25.7517, velocity=-1.0000, tai=1667506455.8453
            1667506455.940: rotation: demandPosition=-25.0300, demandVelocity=-0.0900, demandAcceleration=-0.1200, actualPosition=-25.0300, actualVelocity=-0.0900, debugActualVelocityA=-0.0900, debugActualVelocityB=-0.0800, odometer=392282.8800, timestamp=1667506455.8500
            1667506455.940: motors: raw=[-2452600000.0, -2450500000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
            1667506455.940: ccwFollowingError: positionError=-0.0300, velocityError=-0.0900, timestamp=1667506455.8500
            1667506455.990: rotation: demandPosition=-25.0400, demandVelocity=-0.0900, demandAcceleration=-0.1200, actualPosition=-25.0400, actualVelocity=-0.0900, debugActualVelocityA=-0.0900, debugActualVelocityB=-0.0900, odometer=392282.8900, timestamp=1667506455.9000
            1667506455.990: motors: raw=[-2452500000.0, -2450300000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
            1667506455.990: ccwFollowingError: positionError=-0.0400, velocityError=-0.0900, timestamp=1667506455.9000
            track angle=-25.95 velocity=-1.00
            1667506456.040: rotation: demandPosition=-25.0400, demandVelocity=-0.1000, demandAcceleration=-0.1100, actualPosition=-25.0400, actualVelocity=-0.1000, debugActualVelocityA=-0.1000, debugActualVelocityB=-0.1000, odometer=392282.8900, timestamp=1667506455.9500
            1667506456.040: motors: raw=[-2452300000.0, -2450200000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
            1667506456.041: ccwFollowingError: positionError=-0.0400, velocityError=-0.1000, timestamp=1667506455.9500
            1667506456.090: rotation: demandPosition=-25.0500, demandVelocity=-0.1000, demandAcceleration=-0.0500, actualPosition=-25.0500, actualVelocity=-0.1000, debugActualVelocityA=-0.1000, debugActualVelocityB=-0.1100, odometer=392282.9000, timestamp=1667506456.0000
            1667506456.090: motors: raw=[-2452100000.0, -2450000000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
            1667506456.090: ccwFollowingError: positionError=-0.0500, velocityError=-0.1100, timestamp=1667506456.0000
            1667506456.091: target: position=-25.9516, velocity=-1.0000, tai=1667506456.0452
            1667506456.140: rotation: demandPosition=-25.0500, demandVelocity=-0.1000, demandAcceleration=0.0200, actualPosition=-25.0500, actualVelocity=-0.1000, debugActualVelocityA=-0.1100, debugActualVelocityB=-0.1000, odometer=392282.9000, timestamp=1667506456.0500
            1667506456.140: motors: raw=[-2451900000.0, -2449800000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
            1667506456.140: ccwFollowingError: positionError=-0.0500, velocityError=-0.1000, timestamp=1667506456.0500
            1667506456.190: rotation: demandPosition=-25.0600, demandVelocity=-0.1000, demandAcceleration=0.1600, actualPosition=-25.0600, actualVelocity=-0.1000, debugActualVelocityA=-0.1000, debugActualVelocityB=-0.1000, odometer=392282.9100, timestamp=1667506456.1000
            1667506456.190: motors: raw=[-2451700000.0, -2449600000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
            track angle=-26.00 velocity=0.00
            1667506456.240: rotation: demandPosition=-25.0600, demandVelocity=-0.0900, demandAcceleration=0.3000, actualPosition=-25.0600, actualVelocity=-0.0900, debugActualVelocityA=-0.0900, debugActualVelocityB=-0.1000, odometer=392282.9100, timestamp=1667506456.1500
            1667506456.240: motors: raw=[-2451500000.0, -2449400000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
            1667506456.240: ccwFollowingError: positionError=-0.0600, velocityError=-0.0900, timestamp=1667506456.1500
            1667506456.290: rotation: demandPosition=-25.0600, demandVelocity=-0.0700, demandAcceleration=0.3700, actualPosition=-25.0600, actualVelocity=-0.0700, debugActualVelocityA=-0.0700, debugActualVelocityB=-0.0700, odometer=392282.9200, timestamp=1667506456.2000
            1667506456.290: motors: raw=[-2451400000.0, -2449200000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
            1667506456.291: ccwFollowingError: positionError=-0.0600, velocityError=-0.0700, timestamp=1667506456.2000
            1667506456.290: tracking: tracking=False, lost=True, noNewCommand=False
            1667506456.291: target: position=-26.0000, velocity=0.0000, tai=1667506456.2452
            1667506456.290: inPosition: inPosition=False
            {code}

            This shows an attempt to track a path from -25 to -26 degrees at 1 deg/second (thus a very short ramp). The rotator starts moving in the correct direction but soon turns around. The tracking commands look like this: {{track angle=-25.95 velocity=-1.00}} (though you can also get the same information from the "target: " events, which the CSC publishes when it sends a track command or point-to-point move command to the low-level controller).

            I also attached a graph from [~julen.garcia] showing position vs time (of both rotator and camera cable wrap). This shows the rotator starting off in the correct direction, but then turning around and ending up in the wrong direction from the starting point. Also notice that the end of the motion is very abrupt; I doubt it is properly jerk-limited.
            ttsai Te-Wei Tsai made changes -
            Sprint TSSW Sprint - Dec 19 - Jan 02 [ 1210 ]
            ttsai Te-Wei Tsai made changes -
            Status To Do [ 10001 ] In Progress [ 3 ]
            ttsai Te-Wei Tsai made changes -
            Story Points 0
            ttsai Te-Wei Tsai made changes -
            Reviewers Russell Owen [ rowen ]
            Status In Progress [ 3 ] In Review [ 10004 ]
            aclements Andy Clements made changes -
            Sprint TSSW Sprint - Dec 19 - Jan 02 [ 1210 ] TSSW Sprint - Dec 19 - Jan 02, TSSW Sprint - Jan 02 - Jan 16 [ 1210, 1211 ]
            ttsai Te-Wei Tsai made changes -
            Status In Review [ 10004 ] Reviewed [ 10101 ]
            ttsai Te-Wei Tsai made changes -
            Resolution Done [ 10000 ]
            Status Reviewed [ 10101 ] Done [ 10002 ]

              People

              Assignee:
              ttsai Te-Wei Tsai
              Reporter:
              rowen Russell Owen
              Reviewers:
              Russell Owen
              Watchers:
              Brian Stalder, Bruno Quint, Julen Garcia, Russell Owen, Te-Wei Tsai
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Jenkins

                  No builds found.