Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-36861

The MT rotator is going in the wrong direction when tracking ramps

    XMLWordPrintable

    Details

      Description

      In the last two days every time we try a ramp tracking sequence for the MT rotator from the MTRotator commander, the rotator moves in the wrong direction, often quite far in the wrong direction. Here is an example, with a very short ramp (so I didn't have to stop it):

      MT Rotator failed ramp 2022-11-03

      ramp -25 -26 1
      		 
      run_command running command method <bound method RotatorCommander.do_ramp of <lsst.ts.mtrotator.rotator_commander.RotatorCommander object at 0x7fc1390205e0>>
      		 
      Tracking a ramp from -25.0 to -26.0 at speed 1.0; this will take 1.00 seconds
      		 
      Finished command ramp
      		 
      track angle=-25.00 velocity=-1.00
      		 
      1667506455.090: controllerState: controllerState=2, offlineSubstate=1, enabledSubstate=2, applicationStatus=5124
      		 
      1667506455.090: tracking: tracking=False, lost=True, noNewCommand=False
      		 
      1667506455.091: target: position=-25.0000, velocity=-1.0000, tai=1667506455.0936
      		 
      1667506455.090: inPosition: inPosition=False
      		 
      track angle=-25.15 velocity=-1.00
      		 
      1667506455.240: rotation: demandPosition=-25.0000, demandVelocity=-0.0000, demandAcceleration=-0.0600, actualPosition=-25.0000, actualVelocity=-0.0000, debugActualVelocityA=-0.0000, debugActualVelocityB=-0.0000, odometer=392282.8500, timestamp=1667506455.1500
      		 
      1667506455.240: tracking: tracking=True, lost=True, noNewCommand=False
      		 
      1667506455.240: inPosition: inPosition=True
      		 
      1667506455.290: rotation: demandPosition=-25.0000, demandVelocity=-0.0100, demandAcceleration=-0.1200, actualPosition=-25.0000, actualVelocity=-0.0000, debugActualVelocityA=-0.0000, debugActualVelocityB=-0.0000, odometer=392282.8500, timestamp=1667506455.2000
      		 
      1667506455.290: target: position=-25.1530, velocity=-1.0000, tai=1667506455.2466
      		 
      1667506455.340: rotation: demandPosition=-25.0000, demandVelocity=-0.0100, demandAcceleration=-0.1200, actualPosition=-25.0000, actualVelocity=-0.0200, debugActualVelocityA=-0.0200, debugActualVelocityB=-0.0100, odometer=392282.8500, timestamp=1667506455.2500
      		 
      1667506455.340: ccwFollowingError: positionError=-0.0000, velocityError=-0.0200, timestamp=1667506455.2500
      		 
      1667506455.390: rotation: demandPosition=-25.0000, demandVelocity=-0.0200, demandAcceleration=-0.1200, actualPosition=-25.0000, actualVelocity=-0.0300, debugActualVelocityA=-0.0300, debugActualVelocityB=-0.0300, odometer=392282.8600, timestamp=1667506455.3000
      		 
      1667506455.390: motors: raw=[-2453700000.0, -2451600000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
      		 
      1667506455.390: ccwFollowingError: positionError=-0.0000, velocityError=-0.0300, timestamp=1667506455.3000
      		 
      track angle=-25.35 velocity=-1.00
      		 
      1667506455.440: rotation: demandPosition=-25.0000, demandVelocity=-0.0300, demandAcceleration=-0.1200, actualPosition=-25.0000, actualVelocity=-0.0300, debugActualVelocityA=-0.0300, debugActualVelocityB=-0.0300, odometer=392282.8600, timestamp=1667506455.3500
      		 
      1667506455.490: motors: raw=[-2453600000.0, -2451500000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
      		 
      1667506455.490: target: position=-25.3514, velocity=-1.0000, tai=1667506455.4450
      		 
      1667506455.540: rotation: demandPosition=-25.0100, demandVelocity=-0.0400, demandAcceleration=-0.1200, actualPosition=-25.0100, actualVelocity=-0.0400, debugActualVelocityA=-0.0400, debugActualVelocityB=-0.0400, odometer=392282.8600, timestamp=1667506455.4500
      		 
      1667506455.540: motors: raw=[-2453600000.0, -2451400000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
      		 
      1667506455.540: ccwFollowingError: positionError=-0.0100, velocityError=-0.0400, timestamp=1667506455.4500
      		 
      1667506455.590: motors: raw=[-2453500000.0, -2451400000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
      		 
      track angle=-25.55 velocity=-1.00
      		 
      1667506455.640: rotation: demandPosition=-25.0100, demandVelocity=-0.0500, demandAcceleration=-0.1200, actualPosition=-25.0100, actualVelocity=-0.0500, debugActualVelocityA=-0.0500, debugActualVelocityB=-0.0500, odometer=392282.8600, timestamp=1667506455.5500
      		 
      1667506455.640: motors: raw=[-2453400000.0, -2451300000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
      		 
      1667506455.640: ccwFollowingError: positionError=-0.0100, velocityError=-0.0500, timestamp=1667506455.5500
      		 
      1667506455.690: rotation: demandPosition=-25.0100, demandVelocity=-0.0600, demandAcceleration=-0.1200, actualPosition=-25.0100, actualVelocity=-0.0600, debugActualVelocityA=-0.0600, debugActualVelocityB=-0.0600, odometer=392282.8700, timestamp=1667506455.6000
      		 
      1667506455.690: motors: raw=[-2453300000.0, -2451200000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
      		 
      1667506455.690: ccwFollowingError: positionError=-0.0100, velocityError=-0.0600, timestamp=1667506455.6000
      		 
      1667506455.690: target: position=-25.5512, velocity=-1.0000, tai=1667506455.6448
      		 
      1667506455.740: rotation: demandPosition=-25.0200, demandVelocity=-0.0600, demandAcceleration=-0.1200, actualPosition=-25.0200, actualVelocity=-0.0600, debugActualVelocityA=-0.0600, debugActualVelocityB=-0.0600, odometer=392282.8700, timestamp=1667506455.6500
      		 
      1667506455.740: motors: raw=[-2453200000.0, -2451100000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
      		 
      1667506455.741: ccwFollowingError: positionError=-0.0200, velocityError=-0.0600, timestamp=1667506455.6500
      		 
      1667506455.790: rotation: demandPosition=-25.0200, demandVelocity=-0.0700, demandAcceleration=-0.1200, actualPosition=-25.0200, actualVelocity=-0.0700, debugActualVelocityA=-0.0700, debugActualVelocityB=-0.0700, odometer=392282.8700, timestamp=1667506455.7000
      		 
      1667506455.790: motors: raw=[-2453100000.0, -2451000000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
      		 
      1667506455.791: ccwFollowingError: positionError=-0.0200, velocityError=-0.0700, timestamp=1667506455.7000
      		 
      track angle=-25.75 velocity=-1.00
      		 
      1667506455.840: rotation: demandPosition=-25.0200, demandVelocity=-0.0800, demandAcceleration=-0.1200, actualPosition=-25.0200, actualVelocity=-0.0800, debugActualVelocityA=-0.0800, debugActualVelocityB=-0.0800, odometer=392282.8800, timestamp=1667506455.7500
      		 
      1667506455.840: motors: raw=[-2452900000.0, -2450800000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
      		 
      1667506455.840: ccwFollowingError: positionError=-0.0200, velocityError=-0.0800, timestamp=1667506455.7500
      		 
      1667506455.890: rotation: demandPosition=-25.0300, demandVelocity=-0.0800, demandAcceleration=-0.1200, actualPosition=-25.0300, actualVelocity=-0.0800, debugActualVelocityA=-0.0800, debugActualVelocityB=-0.0800, odometer=392282.8800, timestamp=1667506455.8000
      		 
      1667506455.890: motors: raw=[-2452800000.0, -2450700000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
      		 
      1667506455.890: ccwFollowingError: positionError=-0.0300, velocityError=-0.0800, timestamp=1667506455.8000
      		 
      1667506455.890: target: position=-25.7517, velocity=-1.0000, tai=1667506455.8453
      		 
      1667506455.940: rotation: demandPosition=-25.0300, demandVelocity=-0.0900, demandAcceleration=-0.1200, actualPosition=-25.0300, actualVelocity=-0.0900, debugActualVelocityA=-0.0900, debugActualVelocityB=-0.0800, odometer=392282.8800, timestamp=1667506455.8500
      		 
      1667506455.940: motors: raw=[-2452600000.0, -2450500000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
      		 
      1667506455.940: ccwFollowingError: positionError=-0.0300, velocityError=-0.0900, timestamp=1667506455.8500
      		 
      1667506455.990: rotation: demandPosition=-25.0400, demandVelocity=-0.0900, demandAcceleration=-0.1200, actualPosition=-25.0400, actualVelocity=-0.0900, debugActualVelocityA=-0.0900, debugActualVelocityB=-0.0900, odometer=392282.8900, timestamp=1667506455.9000
      		 
      1667506455.990: motors: raw=[-2452500000.0, -2450300000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
      		 
      1667506455.990: ccwFollowingError: positionError=-0.0400, velocityError=-0.0900, timestamp=1667506455.9000
      		 
      track angle=-25.95 velocity=-1.00
      		 
      1667506456.040: rotation: demandPosition=-25.0400, demandVelocity=-0.1000, demandAcceleration=-0.1100, actualPosition=-25.0400, actualVelocity=-0.1000, debugActualVelocityA=-0.1000, debugActualVelocityB=-0.1000, odometer=392282.8900, timestamp=1667506455.9500
      		 
      1667506456.040: motors: raw=[-2452300000.0, -2450200000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
      		 
      1667506456.041: ccwFollowingError: positionError=-0.0400, velocityError=-0.1000, timestamp=1667506455.9500
      		 
      1667506456.090: rotation: demandPosition=-25.0500, demandVelocity=-0.1000, demandAcceleration=-0.0500, actualPosition=-25.0500, actualVelocity=-0.1000, debugActualVelocityA=-0.1000, debugActualVelocityB=-0.1100, odometer=392282.9000, timestamp=1667506456.0000
      		 
      1667506456.090: motors: raw=[-2452100000.0, -2450000000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
      		 
      1667506456.090: ccwFollowingError: positionError=-0.0500, velocityError=-0.1100, timestamp=1667506456.0000
      		 
      1667506456.091: target: position=-25.9516, velocity=-1.0000, tai=1667506456.0452
      		 
      1667506456.140: rotation: demandPosition=-25.0500, demandVelocity=-0.1000, demandAcceleration=0.0200, actualPosition=-25.0500, actualVelocity=-0.1000, debugActualVelocityA=-0.1100, debugActualVelocityB=-0.1000, odometer=392282.9000, timestamp=1667506456.0500
      		 
      1667506456.140: motors: raw=[-2451900000.0, -2449800000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
      		 
      1667506456.140: ccwFollowingError: positionError=-0.0500, velocityError=-0.1000, timestamp=1667506456.0500
      		 
      1667506456.190: rotation: demandPosition=-25.0600, demandVelocity=-0.1000, demandAcceleration=0.1600, actualPosition=-25.0600, actualVelocity=-0.1000, debugActualVelocityA=-0.1000, debugActualVelocityB=-0.1000, odometer=392282.9100, timestamp=1667506456.1000
      		 
      1667506456.190: motors: raw=[-2451700000.0, -2449600000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
      		 
      track angle=-26.00 velocity=0.00
      		 
      1667506456.240: rotation: demandPosition=-25.0600, demandVelocity=-0.0900, demandAcceleration=0.3000, actualPosition=-25.0600, actualVelocity=-0.0900, debugActualVelocityA=-0.0900, debugActualVelocityB=-0.1000, odometer=392282.9100, timestamp=1667506456.1500
      		 
      1667506456.240: motors: raw=[-2451500000.0, -2449400000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
      		 
      1667506456.240: ccwFollowingError: positionError=-0.0600, velocityError=-0.0900, timestamp=1667506456.1500
      		 
      1667506456.290: rotation: demandPosition=-25.0600, demandVelocity=-0.0700, demandAcceleration=0.3700, actualPosition=-25.0600, actualVelocity=-0.0700, debugActualVelocityA=-0.0700, debugActualVelocityB=-0.0700, odometer=392282.9200, timestamp=1667506456.2000
      		 
      1667506456.290: motors: raw=[-2451400000.0, -2449200000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
      		 
      1667506456.291: ccwFollowingError: positionError=-0.0600, velocityError=-0.0700, timestamp=1667506456.2000
      		 
      1667506456.290: tracking: tracking=False, lost=True, noNewCommand=False
      		 
      1667506456.291: target: position=-26.0000, velocity=0.0000, tai=1667506456.2452
      		 
      1667506456.290: inPosition: inPosition=False

      This shows an attempt to track a path from -25 to -26 degrees at 1 deg/second (thus a very short ramp). The rotator starts moving in the correct direction but soon turns around. The tracking commands look like this: track angle=-25.95 velocity=-1.00 (though you can also get the same information from the "target: " events, which the CSC publishes when it sends a track command or point-to-point move command to the low-level controller).

      I also attached a graph from Julen Garcia showing position vs time (of both rotator and camera cable wrap). This shows the rotator starting off in the correct direction, but then turning around and ending up in the wrong direction from the starting point. Also notice that the end of the motion is very abrupt; I doubt it is properly jerk-limited.

        Attachments

        1. failed ramp.txt
          235 kB
        2. Rotator PosVelAcc.jpg
          Rotator PosVelAcc.jpg
          193 kB
        3. rotator ramp path.png
          rotator ramp path.png
          261 kB

          Issue Links

          relates to

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-36869 The MT rotator can drive itself past the software limits into the direction inhibit limit swiches

          • To Do

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-25245 Put the Rotator into Fault State if no Track Command for a Long Time

          • Done

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-36872 MT rotator TRACK enabled substate command reports done before the rotator is ready to receive TRACK_VEL_CMD commands

          • Done

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-36860 MTRotator low-level controller

          • Invalid

            Activity

            Ascending order - Click to sort in descending order
            No builds found.
            rowen Russell Owen created issue -
            rowen Russell Owen made changes -
            Field Original Value New Value
            Attachment rotator ramp path.png [ 64422 ]
            rowen Russell Owen made changes -
            Description In the last two days every time we try a ramp tracking sequence for the MT rotator from the MTRotator commander, the rotator moves in the wrong direction. Here is an example:

            MT Rotator failed ramp 2022-11-03

            {code}
            ramp -25 -26 1
            run_command running command method <bound method RotatorCommander.do_ramp of <lsst.ts.mtrotator.rotator_commander.RotatorCommander object at 0x7fc1390205e0>>
            Tracking a ramp from -25.0 to -26.0 at speed 1.0; this will take 1.00 seconds
            Finished command ramp
            track angle=-25.00 velocity=-1.00
            1667506455.090: controllerState: controllerState=2, offlineSubstate=1, enabledSubstate=2, applicationStatus=5124
            1667506455.090: tracking: tracking=False, lost=True, noNewCommand=False
            1667506455.091: target: position=-25.0000, velocity=-1.0000, tai=1667506455.0936
            1667506455.090: inPosition: inPosition=False
            track angle=-25.15 velocity=-1.00
            1667506455.240: rotation: demandPosition=-25.0000, demandVelocity=-0.0000, demandAcceleration=-0.0600, actualPosition=-25.0000, actualVelocity=-0.0000, debugActualVelocityA=-0.0000, debugActualVelocityB=-0.0000, odometer=392282.8500, timestamp=1667506455.1500
            1667506455.240: tracking: tracking=True, lost=True, noNewCommand=False
            1667506455.240: inPosition: inPosition=True
            1667506455.290: rotation: demandPosition=-25.0000, demandVelocity=-0.0100, demandAcceleration=-0.1200, actualPosition=-25.0000, actualVelocity=-0.0000, debugActualVelocityA=-0.0000, debugActualVelocityB=-0.0000, odometer=392282.8500, timestamp=1667506455.2000
            1667506455.290: target: position=-25.1530, velocity=-1.0000, tai=1667506455.2466
            1667506455.340: rotation: demandPosition=-25.0000, demandVelocity=-0.0100, demandAcceleration=-0.1200, actualPosition=-25.0000, actualVelocity=-0.0200, debugActualVelocityA=-0.0200, debugActualVelocityB=-0.0100, odometer=392282.8500, timestamp=1667506455.2500
            1667506455.340: ccwFollowingError: positionError=-0.0000, velocityError=-0.0200, timestamp=1667506455.2500
            1667506455.390: rotation: demandPosition=-25.0000, demandVelocity=-0.0200, demandAcceleration=-0.1200, actualPosition=-25.0000, actualVelocity=-0.0300, debugActualVelocityA=-0.0300, debugActualVelocityB=-0.0300, odometer=392282.8600, timestamp=1667506455.3000
            1667506455.390: motors: raw=[-2453700000.0, -2451600000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
            1667506455.390: ccwFollowingError: positionError=-0.0000, velocityError=-0.0300, timestamp=1667506455.3000
            track angle=-25.35 velocity=-1.00
            1667506455.440: rotation: demandPosition=-25.0000, demandVelocity=-0.0300, demandAcceleration=-0.1200, actualPosition=-25.0000, actualVelocity=-0.0300, debugActualVelocityA=-0.0300, debugActualVelocityB=-0.0300, odometer=392282.8600, timestamp=1667506455.3500
            1667506455.490: motors: raw=[-2453600000.0, -2451500000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
            1667506455.490: target: position=-25.3514, velocity=-1.0000, tai=1667506455.4450
            1667506455.540: rotation: demandPosition=-25.0100, demandVelocity=-0.0400, demandAcceleration=-0.1200, actualPosition=-25.0100, actualVelocity=-0.0400, debugActualVelocityA=-0.0400, debugActualVelocityB=-0.0400, odometer=392282.8600, timestamp=1667506455.4500
            1667506455.540: motors: raw=[-2453600000.0, -2451400000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
            1667506455.540: ccwFollowingError: positionError=-0.0100, velocityError=-0.0400, timestamp=1667506455.4500
            1667506455.590: motors: raw=[-2453500000.0, -2451400000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
            track angle=-25.55 velocity=-1.00
            1667506455.640: rotation: demandPosition=-25.0100, demandVelocity=-0.0500, demandAcceleration=-0.1200, actualPosition=-25.0100, actualVelocity=-0.0500, debugActualVelocityA=-0.0500, debugActualVelocityB=-0.0500, odometer=392282.8600, timestamp=1667506455.5500
            1667506455.640: motors: raw=[-2453400000.0, -2451300000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
            1667506455.640: ccwFollowingError: positionError=-0.0100, velocityError=-0.0500, timestamp=1667506455.5500
            1667506455.690: rotation: demandPosition=-25.0100, demandVelocity=-0.0600, demandAcceleration=-0.1200, actualPosition=-25.0100, actualVelocity=-0.0600, debugActualVelocityA=-0.0600, debugActualVelocityB=-0.0600, odometer=392282.8700, timestamp=1667506455.6000
            1667506455.690: motors: raw=[-2453300000.0, -2451200000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
            1667506455.690: ccwFollowingError: positionError=-0.0100, velocityError=-0.0600, timestamp=1667506455.6000
            1667506455.690: target: position=-25.5512, velocity=-1.0000, tai=1667506455.6448
            1667506455.740: rotation: demandPosition=-25.0200, demandVelocity=-0.0600, demandAcceleration=-0.1200, actualPosition=-25.0200, actualVelocity=-0.0600, debugActualVelocityA=-0.0600, debugActualVelocityB=-0.0600, odometer=392282.8700, timestamp=1667506455.6500
            1667506455.740: motors: raw=[-2453200000.0, -2451100000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
            1667506455.741: ccwFollowingError: positionError=-0.0200, velocityError=-0.0600, timestamp=1667506455.6500
            1667506455.790: rotation: demandPosition=-25.0200, demandVelocity=-0.0700, demandAcceleration=-0.1200, actualPosition=-25.0200, actualVelocity=-0.0700, debugActualVelocityA=-0.0700, debugActualVelocityB=-0.0700, odometer=392282.8700, timestamp=1667506455.7000
            1667506455.790: motors: raw=[-2453100000.0, -2451000000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
            1667506455.791: ccwFollowingError: positionError=-0.0200, velocityError=-0.0700, timestamp=1667506455.7000
            track angle=-25.75 velocity=-1.00
            1667506455.840: rotation: demandPosition=-25.0200, demandVelocity=-0.0800, demandAcceleration=-0.1200, actualPosition=-25.0200, actualVelocity=-0.0800, debugActualVelocityA=-0.0800, debugActualVelocityB=-0.0800, odometer=392282.8800, timestamp=1667506455.7500
            1667506455.840: motors: raw=[-2452900000.0, -2450800000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
            1667506455.840: ccwFollowingError: positionError=-0.0200, velocityError=-0.0800, timestamp=1667506455.7500
            1667506455.890: rotation: demandPosition=-25.0300, demandVelocity=-0.0800, demandAcceleration=-0.1200, actualPosition=-25.0300, actualVelocity=-0.0800, debugActualVelocityA=-0.0800, debugActualVelocityB=-0.0800, odometer=392282.8800, timestamp=1667506455.8000
            1667506455.890: motors: raw=[-2452800000.0, -2450700000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
            1667506455.890: ccwFollowingError: positionError=-0.0300, velocityError=-0.0800, timestamp=1667506455.8000
            1667506455.890: target: position=-25.7517, velocity=-1.0000, tai=1667506455.8453
            1667506455.940: rotation: demandPosition=-25.0300, demandVelocity=-0.0900, demandAcceleration=-0.1200, actualPosition=-25.0300, actualVelocity=-0.0900, debugActualVelocityA=-0.0900, debugActualVelocityB=-0.0800, odometer=392282.8800, timestamp=1667506455.8500
            1667506455.940: motors: raw=[-2452600000.0, -2450500000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
            1667506455.940: ccwFollowingError: positionError=-0.0300, velocityError=-0.0900, timestamp=1667506455.8500
            1667506455.990: rotation: demandPosition=-25.0400, demandVelocity=-0.0900, demandAcceleration=-0.1200, actualPosition=-25.0400, actualVelocity=-0.0900, debugActualVelocityA=-0.0900, debugActualVelocityB=-0.0900, odometer=392282.8900, timestamp=1667506455.9000
            1667506455.990: motors: raw=[-2452500000.0, -2450300000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
            1667506455.990: ccwFollowingError: positionError=-0.0400, velocityError=-0.0900, timestamp=1667506455.9000
            track angle=-25.95 velocity=-1.00
            1667506456.040: rotation: demandPosition=-25.0400, demandVelocity=-0.1000, demandAcceleration=-0.1100, actualPosition=-25.0400, actualVelocity=-0.1000, debugActualVelocityA=-0.1000, debugActualVelocityB=-0.1000, odometer=392282.8900, timestamp=1667506455.9500
            1667506456.040: motors: raw=[-2452300000.0, -2450200000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
            1667506456.041: ccwFollowingError: positionError=-0.0400, velocityError=-0.1000, timestamp=1667506455.9500
            1667506456.090: rotation: demandPosition=-25.0500, demandVelocity=-0.1000, demandAcceleration=-0.0500, actualPosition=-25.0500, actualVelocity=-0.1000, debugActualVelocityA=-0.1000, debugActualVelocityB=-0.1100, odometer=392282.9000, timestamp=1667506456.0000
            1667506456.090: motors: raw=[-2452100000.0, -2450000000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
            1667506456.090: ccwFollowingError: positionError=-0.0500, velocityError=-0.1100, timestamp=1667506456.0000
            1667506456.091: target: position=-25.9516, velocity=-1.0000, tai=1667506456.0452
            1667506456.140: rotation: demandPosition=-25.0500, demandVelocity=-0.1000, demandAcceleration=0.0200, actualPosition=-25.0500, actualVelocity=-0.1000, debugActualVelocityA=-0.1100, debugActualVelocityB=-0.1000, odometer=392282.9000, timestamp=1667506456.0500
            1667506456.140: motors: raw=[-2451900000.0, -2449800000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
            1667506456.140: ccwFollowingError: positionError=-0.0500, velocityError=-0.1000, timestamp=1667506456.0500
            1667506456.190: rotation: demandPosition=-25.0600, demandVelocity=-0.1000, demandAcceleration=0.1600, actualPosition=-25.0600, actualVelocity=-0.1000, debugActualVelocityA=-0.1000, debugActualVelocityB=-0.1000, odometer=392282.9100, timestamp=1667506456.1000
            1667506456.190: motors: raw=[-2451700000.0, -2449600000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
            track angle=-26.00 velocity=0.00
            1667506456.240: rotation: demandPosition=-25.0600, demandVelocity=-0.0900, demandAcceleration=0.3000, actualPosition=-25.0600, actualVelocity=-0.0900, debugActualVelocityA=-0.0900, debugActualVelocityB=-0.1000, odometer=392282.9100, timestamp=1667506456.1500
            1667506456.240: motors: raw=[-2451500000.0, -2449400000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
            1667506456.240: ccwFollowingError: positionError=-0.0600, velocityError=-0.0900, timestamp=1667506456.1500
            1667506456.290: rotation: demandPosition=-25.0600, demandVelocity=-0.0700, demandAcceleration=0.3700, actualPosition=-25.0600, actualVelocity=-0.0700, debugActualVelocityA=-0.0700, debugActualVelocityB=-0.0700, odometer=392282.9200, timestamp=1667506456.2000
            1667506456.290: motors: raw=[-2451400000.0, -2449200000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
            1667506456.291: ccwFollowingError: positionError=-0.0600, velocityError=-0.0700, timestamp=1667506456.2000
            1667506456.290: tracking: tracking=False, lost=True, noNewCommand=False
            1667506456.291: target: position=-26.0000, velocity=0.0000, tai=1667506456.2452
            1667506456.290: inPosition: inPosition=False
            {code}

            This shows an attempt to track a path from -25 to -26 degrees at 1 deg/second (thus a very short ramp). The rotator starts moving in the correct direction but soon turns around. The tracking commands look like this: {{track angle=-25.95 velocity=-1.00}}.

            I will also try to attach a graph from [~julen.garcia] showing position vs time (of both rotator and camera cable wrap).             		In the last two days every time we try a ramp tracking sequence for the MT rotator from the MTRotator commander, the rotator moves in the wrong direction, often quite far in the wrong direction. Here is an example, with a very short ramp (so I didn't have to stop it):

            MT Rotator failed ramp 2022-11-03

            {code}
            ramp -25 -26 1
            run_command running command method <bound method RotatorCommander.do_ramp of <lsst.ts.mtrotator.rotator_commander.RotatorCommander object at 0x7fc1390205e0>>
            Tracking a ramp from -25.0 to -26.0 at speed 1.0; this will take 1.00 seconds
            Finished command ramp
            track angle=-25.00 velocity=-1.00
            1667506455.090: controllerState: controllerState=2, offlineSubstate=1, enabledSubstate=2, applicationStatus=5124
            1667506455.090: tracking: tracking=False, lost=True, noNewCommand=False
            1667506455.091: target: position=-25.0000, velocity=-1.0000, tai=1667506455.0936
            1667506455.090: inPosition: inPosition=False
            track angle=-25.15 velocity=-1.00
            1667506455.240: rotation: demandPosition=-25.0000, demandVelocity=-0.0000, demandAcceleration=-0.0600, actualPosition=-25.0000, actualVelocity=-0.0000, debugActualVelocityA=-0.0000, debugActualVelocityB=-0.0000, odometer=392282.8500, timestamp=1667506455.1500
            1667506455.240: tracking: tracking=True, lost=True, noNewCommand=False
            1667506455.240: inPosition: inPosition=True
            1667506455.290: rotation: demandPosition=-25.0000, demandVelocity=-0.0100, demandAcceleration=-0.1200, actualPosition=-25.0000, actualVelocity=-0.0000, debugActualVelocityA=-0.0000, debugActualVelocityB=-0.0000, odometer=392282.8500, timestamp=1667506455.2000
            1667506455.290: target: position=-25.1530, velocity=-1.0000, tai=1667506455.2466
            1667506455.340: rotation: demandPosition=-25.0000, demandVelocity=-0.0100, demandAcceleration=-0.1200, actualPosition=-25.0000, actualVelocity=-0.0200, debugActualVelocityA=-0.0200, debugActualVelocityB=-0.0100, odometer=392282.8500, timestamp=1667506455.2500
            1667506455.340: ccwFollowingError: positionError=-0.0000, velocityError=-0.0200, timestamp=1667506455.2500
            1667506455.390: rotation: demandPosition=-25.0000, demandVelocity=-0.0200, demandAcceleration=-0.1200, actualPosition=-25.0000, actualVelocity=-0.0300, debugActualVelocityA=-0.0300, debugActualVelocityB=-0.0300, odometer=392282.8600, timestamp=1667506455.3000
            1667506455.390: motors: raw=[-2453700000.0, -2451600000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
            1667506455.390: ccwFollowingError: positionError=-0.0000, velocityError=-0.0300, timestamp=1667506455.3000
            track angle=-25.35 velocity=-1.00
            1667506455.440: rotation: demandPosition=-25.0000, demandVelocity=-0.0300, demandAcceleration=-0.1200, actualPosition=-25.0000, actualVelocity=-0.0300, debugActualVelocityA=-0.0300, debugActualVelocityB=-0.0300, odometer=392282.8600, timestamp=1667506455.3500
            1667506455.490: motors: raw=[-2453600000.0, -2451500000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
            1667506455.490: target: position=-25.3514, velocity=-1.0000, tai=1667506455.4450
            1667506455.540: rotation: demandPosition=-25.0100, demandVelocity=-0.0400, demandAcceleration=-0.1200, actualPosition=-25.0100, actualVelocity=-0.0400, debugActualVelocityA=-0.0400, debugActualVelocityB=-0.0400, odometer=392282.8600, timestamp=1667506455.4500
            1667506455.540: motors: raw=[-2453600000.0, -2451400000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
            1667506455.540: ccwFollowingError: positionError=-0.0100, velocityError=-0.0400, timestamp=1667506455.4500
            1667506455.590: motors: raw=[-2453500000.0, -2451400000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
            track angle=-25.55 velocity=-1.00
            1667506455.640: rotation: demandPosition=-25.0100, demandVelocity=-0.0500, demandAcceleration=-0.1200, actualPosition=-25.0100, actualVelocity=-0.0500, debugActualVelocityA=-0.0500, debugActualVelocityB=-0.0500, odometer=392282.8600, timestamp=1667506455.5500
            1667506455.640: motors: raw=[-2453400000.0, -2451300000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
            1667506455.640: ccwFollowingError: positionError=-0.0100, velocityError=-0.0500, timestamp=1667506455.5500
            1667506455.690: rotation: demandPosition=-25.0100, demandVelocity=-0.0600, demandAcceleration=-0.1200, actualPosition=-25.0100, actualVelocity=-0.0600, debugActualVelocityA=-0.0600, debugActualVelocityB=-0.0600, odometer=392282.8700, timestamp=1667506455.6000
            1667506455.690: motors: raw=[-2453300000.0, -2451200000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
            1667506455.690: ccwFollowingError: positionError=-0.0100, velocityError=-0.0600, timestamp=1667506455.6000
            1667506455.690: target: position=-25.5512, velocity=-1.0000, tai=1667506455.6448
            1667506455.740: rotation: demandPosition=-25.0200, demandVelocity=-0.0600, demandAcceleration=-0.1200, actualPosition=-25.0200, actualVelocity=-0.0600, debugActualVelocityA=-0.0600, debugActualVelocityB=-0.0600, odometer=392282.8700, timestamp=1667506455.6500
            1667506455.740: motors: raw=[-2453200000.0, -2451100000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
            1667506455.741: ccwFollowingError: positionError=-0.0200, velocityError=-0.0600, timestamp=1667506455.6500
            1667506455.790: rotation: demandPosition=-25.0200, demandVelocity=-0.0700, demandAcceleration=-0.1200, actualPosition=-25.0200, actualVelocity=-0.0700, debugActualVelocityA=-0.0700, debugActualVelocityB=-0.0700, odometer=392282.8700, timestamp=1667506455.7000
            1667506455.790: motors: raw=[-2453100000.0, -2451000000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
            1667506455.791: ccwFollowingError: positionError=-0.0200, velocityError=-0.0700, timestamp=1667506455.7000
            track angle=-25.75 velocity=-1.00
            1667506455.840: rotation: demandPosition=-25.0200, demandVelocity=-0.0800, demandAcceleration=-0.1200, actualPosition=-25.0200, actualVelocity=-0.0800, debugActualVelocityA=-0.0800, debugActualVelocityB=-0.0800, odometer=392282.8800, timestamp=1667506455.7500
            1667506455.840: motors: raw=[-2452900000.0, -2450800000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
            1667506455.840: ccwFollowingError: positionError=-0.0200, velocityError=-0.0800, timestamp=1667506455.7500
            1667506455.890: rotation: demandPosition=-25.0300, demandVelocity=-0.0800, demandAcceleration=-0.1200, actualPosition=-25.0300, actualVelocity=-0.0800, debugActualVelocityA=-0.0800, debugActualVelocityB=-0.0800, odometer=392282.8800, timestamp=1667506455.8000
            1667506455.890: motors: raw=[-2452800000.0, -2450700000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
            1667506455.890: ccwFollowingError: positionError=-0.0300, velocityError=-0.0800, timestamp=1667506455.8000
            1667506455.890: target: position=-25.7517, velocity=-1.0000, tai=1667506455.8453
            1667506455.940: rotation: demandPosition=-25.0300, demandVelocity=-0.0900, demandAcceleration=-0.1200, actualPosition=-25.0300, actualVelocity=-0.0900, debugActualVelocityA=-0.0900, debugActualVelocityB=-0.0800, odometer=392282.8800, timestamp=1667506455.8500
            1667506455.940: motors: raw=[-2452600000.0, -2450500000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
            1667506455.940: ccwFollowingError: positionError=-0.0300, velocityError=-0.0900, timestamp=1667506455.8500
            1667506455.990: rotation: demandPosition=-25.0400, demandVelocity=-0.0900, demandAcceleration=-0.1200, actualPosition=-25.0400, actualVelocity=-0.0900, debugActualVelocityA=-0.0900, debugActualVelocityB=-0.0900, odometer=392282.8900, timestamp=1667506455.9000
            1667506455.990: motors: raw=[-2452500000.0, -2450300000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
            1667506455.990: ccwFollowingError: positionError=-0.0400, velocityError=-0.0900, timestamp=1667506455.9000
            track angle=-25.95 velocity=-1.00
            1667506456.040: rotation: demandPosition=-25.0400, demandVelocity=-0.1000, demandAcceleration=-0.1100, actualPosition=-25.0400, actualVelocity=-0.1000, debugActualVelocityA=-0.1000, debugActualVelocityB=-0.1000, odometer=392282.8900, timestamp=1667506455.9500
            1667506456.040: motors: raw=[-2452300000.0, -2450200000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
            1667506456.041: ccwFollowingError: positionError=-0.0400, velocityError=-0.1000, timestamp=1667506455.9500
            1667506456.090: rotation: demandPosition=-25.0500, demandVelocity=-0.1000, demandAcceleration=-0.0500, actualPosition=-25.0500, actualVelocity=-0.1000, debugActualVelocityA=-0.1000, debugActualVelocityB=-0.1100, odometer=392282.9000, timestamp=1667506456.0000
            1667506456.090: motors: raw=[-2452100000.0, -2450000000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
            1667506456.090: ccwFollowingError: positionError=-0.0500, velocityError=-0.1100, timestamp=1667506456.0000
            1667506456.091: target: position=-25.9516, velocity=-1.0000, tai=1667506456.0452
            1667506456.140: rotation: demandPosition=-25.0500, demandVelocity=-0.1000, demandAcceleration=0.0200, actualPosition=-25.0500, actualVelocity=-0.1000, debugActualVelocityA=-0.1100, debugActualVelocityB=-0.1000, odometer=392282.9000, timestamp=1667506456.0500
            1667506456.140: motors: raw=[-2451900000.0, -2449800000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
            1667506456.140: ccwFollowingError: positionError=-0.0500, velocityError=-0.1000, timestamp=1667506456.0500
            1667506456.190: rotation: demandPosition=-25.0600, demandVelocity=-0.1000, demandAcceleration=0.1600, actualPosition=-25.0600, actualVelocity=-0.1000, debugActualVelocityA=-0.1000, debugActualVelocityB=-0.1000, odometer=392282.9100, timestamp=1667506456.1000
            1667506456.190: motors: raw=[-2451700000.0, -2449600000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
            track angle=-26.00 velocity=0.00
            1667506456.240: rotation: demandPosition=-25.0600, demandVelocity=-0.0900, demandAcceleration=0.3000, actualPosition=-25.0600, actualVelocity=-0.0900, debugActualVelocityA=-0.0900, debugActualVelocityB=-0.1000, odometer=392282.9100, timestamp=1667506456.1500
            1667506456.240: motors: raw=[-2451500000.0, -2449400000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
            1667506456.240: ccwFollowingError: positionError=-0.0600, velocityError=-0.0900, timestamp=1667506456.1500
            1667506456.290: rotation: demandPosition=-25.0600, demandVelocity=-0.0700, demandAcceleration=0.3700, actualPosition=-25.0600, actualVelocity=-0.0700, debugActualVelocityA=-0.0700, debugActualVelocityB=-0.0700, odometer=392282.9200, timestamp=1667506456.2000
            1667506456.290: motors: raw=[-2451400000.0, -2449200000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
            1667506456.291: ccwFollowingError: positionError=-0.0600, velocityError=-0.0700, timestamp=1667506456.2000
            1667506456.290: tracking: tracking=False, lost=True, noNewCommand=False
            1667506456.291: target: position=-26.0000, velocity=0.0000, tai=1667506456.2452
            1667506456.290: inPosition: inPosition=False
            {code}

            This shows an attempt to track a path from -25 to -26 degrees at 1 deg/second (thus a very short ramp). The rotator starts moving in the correct direction but soon turns around. The tracking commands look like this: {{track angle=-25.95 velocity=-1.00}}.

            I will also try to attach a graph from [~julen.garcia] showing position vs time (of both rotator and camera cable wrap).
            ttsai Te-Wei Tsai made changes -
            Epic Link DM-27721 [ 442128 ]
            Labels Rotator
            ttsai Te-Wei Tsai made changes -
            Link This issue relates to DM-36860 [ DM-36860 ]
            Hide
            Permalink
            ttsai Te-Wei Tsai added a comment -

            Looks like the bug in trajectory algorithm.

            Show
            ttsai Te-Wei Tsai added a comment - Looks like the bug in trajectory algorithm.
            bquint Bruno Quint made changes -
            Link This issue relates to FRACAS-129 [ FRACAS-129 ]
            bquint Bruno Quint made changes -
            Link This issue relates to FRACAS-129 [ FRACAS-129 ]
            bquint Bruno Quint made changes -
            Link This issue relates to FRACAS-129 [ FRACAS-129 ]
            Hide
            Permalink
            rowen Russell Owen added a comment - - edited

            I agree.

            I tried some more ramps on 2022-11-04 and found the following:

            1: The ramps would not start until I put a 0.1 second delay after the "trackStart" (enter tracking mode) command was finished, before sending the first tracking position. See DM-36872.

            2: I attached a log showing two ramps, one successful, one that went the wrong direction.

            The first ramp at 0.1 deg/sec seemed to work well.

            The second ramp at 1 deg/sec clearly misbehaved:

            • The rotator started out moving in the correct direction, then turned around and went far in the wrong direction.
            • No part of the track is at the requested 1 deg/second; the rotator is either not moving or else is accelerating hard.
            • There is no sign of jerk limiting.
            Show
            rowen Russell Owen added a comment - - edited I agree. I tried some more ramps on 2022-11-04 and found the following: 1: The ramps would not start until I put a 0.1 second delay after the "trackStart" (enter tracking mode) command was finished, before sending the first tracking position. See DM-36872 . 2: I attached a log showing two ramps, one successful, one that went the wrong direction. The first ramp at 0.1 deg/sec seemed to work well. The second ramp at 1 deg/sec clearly misbehaved: The rotator started out moving in the correct direction, then turned around and went far in the wrong direction. No part of the track is at the requested 1 deg/second; the rotator is either not moving or else is accelerating hard. There is no sign of jerk limiting.
            rowen Russell Owen made changes -
            Description In the last two days every time we try a ramp tracking sequence for the MT rotator from the MTRotator commander, the rotator moves in the wrong direction, often quite far in the wrong direction. Here is an example, with a very short ramp (so I didn't have to stop it):

            MT Rotator failed ramp 2022-11-03

            {code}
            ramp -25 -26 1
            run_command running command method <bound method RotatorCommander.do_ramp of <lsst.ts.mtrotator.rotator_commander.RotatorCommander object at 0x7fc1390205e0>>
            Tracking a ramp from -25.0 to -26.0 at speed 1.0; this will take 1.00 seconds
            Finished command ramp
            track angle=-25.00 velocity=-1.00
            1667506455.090: controllerState: controllerState=2, offlineSubstate=1, enabledSubstate=2, applicationStatus=5124
            1667506455.090: tracking: tracking=False, lost=True, noNewCommand=False
            1667506455.091: target: position=-25.0000, velocity=-1.0000, tai=1667506455.0936
            1667506455.090: inPosition: inPosition=False
            track angle=-25.15 velocity=-1.00
            1667506455.240: rotation: demandPosition=-25.0000, demandVelocity=-0.0000, demandAcceleration=-0.0600, actualPosition=-25.0000, actualVelocity=-0.0000, debugActualVelocityA=-0.0000, debugActualVelocityB=-0.0000, odometer=392282.8500, timestamp=1667506455.1500
            1667506455.240: tracking: tracking=True, lost=True, noNewCommand=False
            1667506455.240: inPosition: inPosition=True
            1667506455.290: rotation: demandPosition=-25.0000, demandVelocity=-0.0100, demandAcceleration=-0.1200, actualPosition=-25.0000, actualVelocity=-0.0000, debugActualVelocityA=-0.0000, debugActualVelocityB=-0.0000, odometer=392282.8500, timestamp=1667506455.2000
            1667506455.290: target: position=-25.1530, velocity=-1.0000, tai=1667506455.2466
            1667506455.340: rotation: demandPosition=-25.0000, demandVelocity=-0.0100, demandAcceleration=-0.1200, actualPosition=-25.0000, actualVelocity=-0.0200, debugActualVelocityA=-0.0200, debugActualVelocityB=-0.0100, odometer=392282.8500, timestamp=1667506455.2500
            1667506455.340: ccwFollowingError: positionError=-0.0000, velocityError=-0.0200, timestamp=1667506455.2500
            1667506455.390: rotation: demandPosition=-25.0000, demandVelocity=-0.0200, demandAcceleration=-0.1200, actualPosition=-25.0000, actualVelocity=-0.0300, debugActualVelocityA=-0.0300, debugActualVelocityB=-0.0300, odometer=392282.8600, timestamp=1667506455.3000
            1667506455.390: motors: raw=[-2453700000.0, -2451600000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
            1667506455.390: ccwFollowingError: positionError=-0.0000, velocityError=-0.0300, timestamp=1667506455.3000
            track angle=-25.35 velocity=-1.00
            1667506455.440: rotation: demandPosition=-25.0000, demandVelocity=-0.0300, demandAcceleration=-0.1200, actualPosition=-25.0000, actualVelocity=-0.0300, debugActualVelocityA=-0.0300, debugActualVelocityB=-0.0300, odometer=392282.8600, timestamp=1667506455.3500
            1667506455.490: motors: raw=[-2453600000.0, -2451500000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
            1667506455.490: target: position=-25.3514, velocity=-1.0000, tai=1667506455.4450
            1667506455.540: rotation: demandPosition=-25.0100, demandVelocity=-0.0400, demandAcceleration=-0.1200, actualPosition=-25.0100, actualVelocity=-0.0400, debugActualVelocityA=-0.0400, debugActualVelocityB=-0.0400, odometer=392282.8600, timestamp=1667506455.4500
            1667506455.540: motors: raw=[-2453600000.0, -2451400000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
            1667506455.540: ccwFollowingError: positionError=-0.0100, velocityError=-0.0400, timestamp=1667506455.4500
            1667506455.590: motors: raw=[-2453500000.0, -2451400000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
            track angle=-25.55 velocity=-1.00
            1667506455.640: rotation: demandPosition=-25.0100, demandVelocity=-0.0500, demandAcceleration=-0.1200, actualPosition=-25.0100, actualVelocity=-0.0500, debugActualVelocityA=-0.0500, debugActualVelocityB=-0.0500, odometer=392282.8600, timestamp=1667506455.5500
            1667506455.640: motors: raw=[-2453400000.0, -2451300000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
            1667506455.640: ccwFollowingError: positionError=-0.0100, velocityError=-0.0500, timestamp=1667506455.5500
            1667506455.690: rotation: demandPosition=-25.0100, demandVelocity=-0.0600, demandAcceleration=-0.1200, actualPosition=-25.0100, actualVelocity=-0.0600, debugActualVelocityA=-0.0600, debugActualVelocityB=-0.0600, odometer=392282.8700, timestamp=1667506455.6000
            1667506455.690: motors: raw=[-2453300000.0, -2451200000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
            1667506455.690: ccwFollowingError: positionError=-0.0100, velocityError=-0.0600, timestamp=1667506455.6000
            1667506455.690: target: position=-25.5512, velocity=-1.0000, tai=1667506455.6448
            1667506455.740: rotation: demandPosition=-25.0200, demandVelocity=-0.0600, demandAcceleration=-0.1200, actualPosition=-25.0200, actualVelocity=-0.0600, debugActualVelocityA=-0.0600, debugActualVelocityB=-0.0600, odometer=392282.8700, timestamp=1667506455.6500
            1667506455.740: motors: raw=[-2453200000.0, -2451100000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
            1667506455.741: ccwFollowingError: positionError=-0.0200, velocityError=-0.0600, timestamp=1667506455.6500
            1667506455.790: rotation: demandPosition=-25.0200, demandVelocity=-0.0700, demandAcceleration=-0.1200, actualPosition=-25.0200, actualVelocity=-0.0700, debugActualVelocityA=-0.0700, debugActualVelocityB=-0.0700, odometer=392282.8700, timestamp=1667506455.7000
            1667506455.790: motors: raw=[-2453100000.0, -2451000000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
            1667506455.791: ccwFollowingError: positionError=-0.0200, velocityError=-0.0700, timestamp=1667506455.7000
            track angle=-25.75 velocity=-1.00
            1667506455.840: rotation: demandPosition=-25.0200, demandVelocity=-0.0800, demandAcceleration=-0.1200, actualPosition=-25.0200, actualVelocity=-0.0800, debugActualVelocityA=-0.0800, debugActualVelocityB=-0.0800, odometer=392282.8800, timestamp=1667506455.7500
            1667506455.840: motors: raw=[-2452900000.0, -2450800000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
            1667506455.840: ccwFollowingError: positionError=-0.0200, velocityError=-0.0800, timestamp=1667506455.7500
            1667506455.890: rotation: demandPosition=-25.0300, demandVelocity=-0.0800, demandAcceleration=-0.1200, actualPosition=-25.0300, actualVelocity=-0.0800, debugActualVelocityA=-0.0800, debugActualVelocityB=-0.0800, odometer=392282.8800, timestamp=1667506455.8000
            1667506455.890: motors: raw=[-2452800000.0, -2450700000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
            1667506455.890: ccwFollowingError: positionError=-0.0300, velocityError=-0.0800, timestamp=1667506455.8000
            1667506455.890: target: position=-25.7517, velocity=-1.0000, tai=1667506455.8453
            1667506455.940: rotation: demandPosition=-25.0300, demandVelocity=-0.0900, demandAcceleration=-0.1200, actualPosition=-25.0300, actualVelocity=-0.0900, debugActualVelocityA=-0.0900, debugActualVelocityB=-0.0800, odometer=392282.8800, timestamp=1667506455.8500
            1667506455.940: motors: raw=[-2452600000.0, -2450500000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
            1667506455.940: ccwFollowingError: positionError=-0.0300, velocityError=-0.0900, timestamp=1667506455.8500
            1667506455.990: rotation: demandPosition=-25.0400, demandVelocity=-0.0900, demandAcceleration=-0.1200, actualPosition=-25.0400, actualVelocity=-0.0900, debugActualVelocityA=-0.0900, debugActualVelocityB=-0.0900, odometer=392282.8900, timestamp=1667506455.9000
            1667506455.990: motors: raw=[-2452500000.0, -2450300000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
            1667506455.990: ccwFollowingError: positionError=-0.0400, velocityError=-0.0900, timestamp=1667506455.9000
            track angle=-25.95 velocity=-1.00
            1667506456.040: rotation: demandPosition=-25.0400, demandVelocity=-0.1000, demandAcceleration=-0.1100, actualPosition=-25.0400, actualVelocity=-0.1000, debugActualVelocityA=-0.1000, debugActualVelocityB=-0.1000, odometer=392282.8900, timestamp=1667506455.9500
            1667506456.040: motors: raw=[-2452300000.0, -2450200000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
            1667506456.041: ccwFollowingError: positionError=-0.0400, velocityError=-0.1000, timestamp=1667506455.9500
            1667506456.090: rotation: demandPosition=-25.0500, demandVelocity=-0.1000, demandAcceleration=-0.0500, actualPosition=-25.0500, actualVelocity=-0.1000, debugActualVelocityA=-0.1000, debugActualVelocityB=-0.1100, odometer=392282.9000, timestamp=1667506456.0000
            1667506456.090: motors: raw=[-2452100000.0, -2450000000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
            1667506456.090: ccwFollowingError: positionError=-0.0500, velocityError=-0.1100, timestamp=1667506456.0000
            1667506456.091: target: position=-25.9516, velocity=-1.0000, tai=1667506456.0452
            1667506456.140: rotation: demandPosition=-25.0500, demandVelocity=-0.1000, demandAcceleration=0.0200, actualPosition=-25.0500, actualVelocity=-0.1000, debugActualVelocityA=-0.1100, debugActualVelocityB=-0.1000, odometer=392282.9000, timestamp=1667506456.0500
            1667506456.140: motors: raw=[-2451900000.0, -2449800000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
            1667506456.140: ccwFollowingError: positionError=-0.0500, velocityError=-0.1000, timestamp=1667506456.0500
            1667506456.190: rotation: demandPosition=-25.0600, demandVelocity=-0.1000, demandAcceleration=0.1600, actualPosition=-25.0600, actualVelocity=-0.1000, debugActualVelocityA=-0.1000, debugActualVelocityB=-0.1000, odometer=392282.9100, timestamp=1667506456.1000
            1667506456.190: motors: raw=[-2451700000.0, -2449600000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
            track angle=-26.00 velocity=0.00
            1667506456.240: rotation: demandPosition=-25.0600, demandVelocity=-0.0900, demandAcceleration=0.3000, actualPosition=-25.0600, actualVelocity=-0.0900, debugActualVelocityA=-0.0900, debugActualVelocityB=-0.1000, odometer=392282.9100, timestamp=1667506456.1500
            1667506456.240: motors: raw=[-2451500000.0, -2449400000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
            1667506456.240: ccwFollowingError: positionError=-0.0600, velocityError=-0.0900, timestamp=1667506456.1500
            1667506456.290: rotation: demandPosition=-25.0600, demandVelocity=-0.0700, demandAcceleration=0.3700, actualPosition=-25.0600, actualVelocity=-0.0700, debugActualVelocityA=-0.0700, debugActualVelocityB=-0.0700, odometer=392282.9200, timestamp=1667506456.2000
            1667506456.290: motors: raw=[-2451400000.0, -2449200000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
            1667506456.291: ccwFollowingError: positionError=-0.0600, velocityError=-0.0700, timestamp=1667506456.2000
            1667506456.290: tracking: tracking=False, lost=True, noNewCommand=False
            1667506456.291: target: position=-26.0000, velocity=0.0000, tai=1667506456.2452
            1667506456.290: inPosition: inPosition=False
            {code}

            This shows an attempt to track a path from -25 to -26 degrees at 1 deg/second (thus a very short ramp). The rotator starts moving in the correct direction but soon turns around. The tracking commands look like this: {{track angle=-25.95 velocity=-1.00}}.

            I will also try to attach a graph from [~julen.garcia] showing position vs time (of both rotator and camera cable wrap).             		In the last two days every time we try a ramp tracking sequence for the MT rotator from the MTRotator commander, the rotator moves in the wrong direction, often quite far in the wrong direction. Here is an example, with a very short ramp (so I didn't have to stop it):

            MT Rotator failed ramp 2022-11-03

            {code}
            ramp -25 -26 1
            run_command running command method <bound method RotatorCommander.do_ramp of <lsst.ts.mtrotator.rotator_commander.RotatorCommander object at 0x7fc1390205e0>>
            Tracking a ramp from -25.0 to -26.0 at speed 1.0; this will take 1.00 seconds
            Finished command ramp
            track angle=-25.00 velocity=-1.00
            1667506455.090: controllerState: controllerState=2, offlineSubstate=1, enabledSubstate=2, applicationStatus=5124
            1667506455.090: tracking: tracking=False, lost=True, noNewCommand=False
            1667506455.091: target: position=-25.0000, velocity=-1.0000, tai=1667506455.0936
            1667506455.090: inPosition: inPosition=False
            track angle=-25.15 velocity=-1.00
            1667506455.240: rotation: demandPosition=-25.0000, demandVelocity=-0.0000, demandAcceleration=-0.0600, actualPosition=-25.0000, actualVelocity=-0.0000, debugActualVelocityA=-0.0000, debugActualVelocityB=-0.0000, odometer=392282.8500, timestamp=1667506455.1500
            1667506455.240: tracking: tracking=True, lost=True, noNewCommand=False
            1667506455.240: inPosition: inPosition=True
            1667506455.290: rotation: demandPosition=-25.0000, demandVelocity=-0.0100, demandAcceleration=-0.1200, actualPosition=-25.0000, actualVelocity=-0.0000, debugActualVelocityA=-0.0000, debugActualVelocityB=-0.0000, odometer=392282.8500, timestamp=1667506455.2000
            1667506455.290: target: position=-25.1530, velocity=-1.0000, tai=1667506455.2466
            1667506455.340: rotation: demandPosition=-25.0000, demandVelocity=-0.0100, demandAcceleration=-0.1200, actualPosition=-25.0000, actualVelocity=-0.0200, debugActualVelocityA=-0.0200, debugActualVelocityB=-0.0100, odometer=392282.8500, timestamp=1667506455.2500
            1667506455.340: ccwFollowingError: positionError=-0.0000, velocityError=-0.0200, timestamp=1667506455.2500
            1667506455.390: rotation: demandPosition=-25.0000, demandVelocity=-0.0200, demandAcceleration=-0.1200, actualPosition=-25.0000, actualVelocity=-0.0300, debugActualVelocityA=-0.0300, debugActualVelocityB=-0.0300, odometer=392282.8600, timestamp=1667506455.3000
            1667506455.390: motors: raw=[-2453700000.0, -2451600000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
            1667506455.390: ccwFollowingError: positionError=-0.0000, velocityError=-0.0300, timestamp=1667506455.3000
            track angle=-25.35 velocity=-1.00
            1667506455.440: rotation: demandPosition=-25.0000, demandVelocity=-0.0300, demandAcceleration=-0.1200, actualPosition=-25.0000, actualVelocity=-0.0300, debugActualVelocityA=-0.0300, debugActualVelocityB=-0.0300, odometer=392282.8600, timestamp=1667506455.3500
            1667506455.490: motors: raw=[-2453600000.0, -2451500000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
            1667506455.490: target: position=-25.3514, velocity=-1.0000, tai=1667506455.4450
            1667506455.540: rotation: demandPosition=-25.0100, demandVelocity=-0.0400, demandAcceleration=-0.1200, actualPosition=-25.0100, actualVelocity=-0.0400, debugActualVelocityA=-0.0400, debugActualVelocityB=-0.0400, odometer=392282.8600, timestamp=1667506455.4500
            1667506455.540: motors: raw=[-2453600000.0, -2451400000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
            1667506455.540: ccwFollowingError: positionError=-0.0100, velocityError=-0.0400, timestamp=1667506455.4500
            1667506455.590: motors: raw=[-2453500000.0, -2451400000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
            track angle=-25.55 velocity=-1.00
            1667506455.640: rotation: demandPosition=-25.0100, demandVelocity=-0.0500, demandAcceleration=-0.1200, actualPosition=-25.0100, actualVelocity=-0.0500, debugActualVelocityA=-0.0500, debugActualVelocityB=-0.0500, odometer=392282.8600, timestamp=1667506455.5500
            1667506455.640: motors: raw=[-2453400000.0, -2451300000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
            1667506455.640: ccwFollowingError: positionError=-0.0100, velocityError=-0.0500, timestamp=1667506455.5500
            1667506455.690: rotation: demandPosition=-25.0100, demandVelocity=-0.0600, demandAcceleration=-0.1200, actualPosition=-25.0100, actualVelocity=-0.0600, debugActualVelocityA=-0.0600, debugActualVelocityB=-0.0600, odometer=392282.8700, timestamp=1667506455.6000
            1667506455.690: motors: raw=[-2453300000.0, -2451200000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
            1667506455.690: ccwFollowingError: positionError=-0.0100, velocityError=-0.0600, timestamp=1667506455.6000
            1667506455.690: target: position=-25.5512, velocity=-1.0000, tai=1667506455.6448
            1667506455.740: rotation: demandPosition=-25.0200, demandVelocity=-0.0600, demandAcceleration=-0.1200, actualPosition=-25.0200, actualVelocity=-0.0600, debugActualVelocityA=-0.0600, debugActualVelocityB=-0.0600, odometer=392282.8700, timestamp=1667506455.6500
            1667506455.740: motors: raw=[-2453200000.0, -2451100000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
            1667506455.741: ccwFollowingError: positionError=-0.0200, velocityError=-0.0600, timestamp=1667506455.6500
            1667506455.790: rotation: demandPosition=-25.0200, demandVelocity=-0.0700, demandAcceleration=-0.1200, actualPosition=-25.0200, actualVelocity=-0.0700, debugActualVelocityA=-0.0700, debugActualVelocityB=-0.0700, odometer=392282.8700, timestamp=1667506455.7000
            1667506455.790: motors: raw=[-2453100000.0, -2451000000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
            1667506455.791: ccwFollowingError: positionError=-0.0200, velocityError=-0.0700, timestamp=1667506455.7000
            track angle=-25.75 velocity=-1.00
            1667506455.840: rotation: demandPosition=-25.0200, demandVelocity=-0.0800, demandAcceleration=-0.1200, actualPosition=-25.0200, actualVelocity=-0.0800, debugActualVelocityA=-0.0800, debugActualVelocityB=-0.0800, odometer=392282.8800, timestamp=1667506455.7500
            1667506455.840: motors: raw=[-2452900000.0, -2450800000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
            1667506455.840: ccwFollowingError: positionError=-0.0200, velocityError=-0.0800, timestamp=1667506455.7500
            1667506455.890: rotation: demandPosition=-25.0300, demandVelocity=-0.0800, demandAcceleration=-0.1200, actualPosition=-25.0300, actualVelocity=-0.0800, debugActualVelocityA=-0.0800, debugActualVelocityB=-0.0800, odometer=392282.8800, timestamp=1667506455.8000
            1667506455.890: motors: raw=[-2452800000.0, -2450700000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
            1667506455.890: ccwFollowingError: positionError=-0.0300, velocityError=-0.0800, timestamp=1667506455.8000
            1667506455.890: target: position=-25.7517, velocity=-1.0000, tai=1667506455.8453
            1667506455.940: rotation: demandPosition=-25.0300, demandVelocity=-0.0900, demandAcceleration=-0.1200, actualPosition=-25.0300, actualVelocity=-0.0900, debugActualVelocityA=-0.0900, debugActualVelocityB=-0.0800, odometer=392282.8800, timestamp=1667506455.8500
            1667506455.940: motors: raw=[-2452600000.0, -2450500000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
            1667506455.940: ccwFollowingError: positionError=-0.0300, velocityError=-0.0900, timestamp=1667506455.8500
            1667506455.990: rotation: demandPosition=-25.0400, demandVelocity=-0.0900, demandAcceleration=-0.1200, actualPosition=-25.0400, actualVelocity=-0.0900, debugActualVelocityA=-0.0900, debugActualVelocityB=-0.0900, odometer=392282.8900, timestamp=1667506455.9000
            1667506455.990: motors: raw=[-2452500000.0, -2450300000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
            1667506455.990: ccwFollowingError: positionError=-0.0400, velocityError=-0.0900, timestamp=1667506455.9000
            track angle=-25.95 velocity=-1.00
            1667506456.040: rotation: demandPosition=-25.0400, demandVelocity=-0.1000, demandAcceleration=-0.1100, actualPosition=-25.0400, actualVelocity=-0.1000, debugActualVelocityA=-0.1000, debugActualVelocityB=-0.1000, odometer=392282.8900, timestamp=1667506455.9500
            1667506456.040: motors: raw=[-2452300000.0, -2450200000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
            1667506456.041: ccwFollowingError: positionError=-0.0400, velocityError=-0.1000, timestamp=1667506455.9500
            1667506456.090: rotation: demandPosition=-25.0500, demandVelocity=-0.1000, demandAcceleration=-0.0500, actualPosition=-25.0500, actualVelocity=-0.1000, debugActualVelocityA=-0.1000, debugActualVelocityB=-0.1100, odometer=392282.9000, timestamp=1667506456.0000
            1667506456.090: motors: raw=[-2452100000.0, -2450000000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
            1667506456.090: ccwFollowingError: positionError=-0.0500, velocityError=-0.1100, timestamp=1667506456.0000
            1667506456.091: target: position=-25.9516, velocity=-1.0000, tai=1667506456.0452
            1667506456.140: rotation: demandPosition=-25.0500, demandVelocity=-0.1000, demandAcceleration=0.0200, actualPosition=-25.0500, actualVelocity=-0.1000, debugActualVelocityA=-0.1100, debugActualVelocityB=-0.1000, odometer=392282.9000, timestamp=1667506456.0500
            1667506456.140: motors: raw=[-2451900000.0, -2449800000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
            1667506456.140: ccwFollowingError: positionError=-0.0500, velocityError=-0.1000, timestamp=1667506456.0500
            1667506456.190: rotation: demandPosition=-25.0600, demandVelocity=-0.1000, demandAcceleration=0.1600, actualPosition=-25.0600, actualVelocity=-0.1000, debugActualVelocityA=-0.1000, debugActualVelocityB=-0.1000, odometer=392282.9100, timestamp=1667506456.1000
            1667506456.190: motors: raw=[-2451700000.0, -2449600000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
            track angle=-26.00 velocity=0.00
            1667506456.240: rotation: demandPosition=-25.0600, demandVelocity=-0.0900, demandAcceleration=0.3000, actualPosition=-25.0600, actualVelocity=-0.0900, debugActualVelocityA=-0.0900, debugActualVelocityB=-0.1000, odometer=392282.9100, timestamp=1667506456.1500
            1667506456.240: motors: raw=[-2451500000.0, -2449400000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
            1667506456.240: ccwFollowingError: positionError=-0.0600, velocityError=-0.0900, timestamp=1667506456.1500
            1667506456.290: rotation: demandPosition=-25.0600, demandVelocity=-0.0700, demandAcceleration=0.3700, actualPosition=-25.0600, actualVelocity=-0.0700, debugActualVelocityA=-0.0700, debugActualVelocityB=-0.0700, odometer=392282.9200, timestamp=1667506456.2000
            1667506456.290: motors: raw=[-2451400000.0, -2449200000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
            1667506456.291: ccwFollowingError: positionError=-0.0600, velocityError=-0.0700, timestamp=1667506456.2000
            1667506456.290: tracking: tracking=False, lost=True, noNewCommand=False
            1667506456.291: target: position=-26.0000, velocity=0.0000, tai=1667506456.2452
            1667506456.290: inPosition: inPosition=False
            {code}

            This shows an attempt to track a path from -25 to -26 degrees at 1 deg/second (thus a very short ramp). The rotator starts moving in the correct direction but soon turns around. The tracking commands look like this: {{track angle=-25.95 velocity=-1.00}} (though you can also get the same information from the "target: " events, which the CSC publishes when it sends a track command or point-to-point move command to the low-level controller).

            I also attached a graph from [~julen.garcia] showing position vs time (of both rotator and camera cable wrap). This shows the rotator starting off in the correct direction, but then turning around and ending up in the wrong direction from the starting point. Also notice that the end of the motion is very abrupt; I doubt it is properly jerk-limited.
            rowen Russell Owen made changes -
            Link This issue relates to DM-36869 [ DM-36869 ]
            rowen Russell Owen made changes -
            Attachment failed ramp.txt [ 64440 ]
            rowen Russell Owen made changes -
            Link This issue relates to DM-36872 [ DM-36872 ]
            ttsai Te-Wei Tsai made changes -
            Link This issue relates to DM-25245 [ DM-25245 ]
            Hide
            Permalink
            ttsai Te-Wei Tsai added a comment - - edited

            Russell Owen, the delay time was requested to be hard-coded to be 0.25 second (5 lost tracking commands) at DM-25245. What is the time do you want now? Another option is to put it into the configuration file. Which way do you prefer?

            Show
            ttsai Te-Wei Tsai added a comment - - edited Russell Owen , the delay time was requested to be hard-coded to be 0.25 second (5 lost tracking commands) at DM-25245 . What is the time do you want now? Another option is to put it into the configuration file. Which way do you prefer?
            bquint Bruno Quint made changes -
            Attachment Rotator PosVelAcc.jpg [ 64469 ]
            Hide
            Permalink
            bquint Bruno Quint added a comment -

            I attached a plot containing the demand/actual/error values for the position, velocity, and acceleration. The actual acceleration was calculated using demandVelocity gradient and the time difference between each row.

            Show
            bquint Bruno Quint added a comment - I attached a plot containing the demand/actual/error values for the position, velocity, and acceleration. The actual acceleration was calculated using demandVelocity gradient and the time difference between each row.
            bquint Bruno Quint made changes -
            Attachment Rotator PosVelAcc.jpg [ 64469 ]
            bquint Bruno Quint made changes -
            Attachment Rotator PosVelAcc.jpg [ 64470 ]
            rowen Russell Owen made changes -
            Watchers Brian Stalder, Bruno Quint, Russell Owen, Te-Wei Tsai [ Brian Stalder, Bruno Quint, Russell Owen, Te-Wei Tsai ] Brian Stalder, Bruno Quint, Julen Garcia, Russell Owen, Te-Wei Tsai [ Brian Stalder, Bruno Quint, Julen Garcia, Russell Owen, Te-Wei Tsai ]
            rowen Russell Owen made changes -
            Description In the last two days every time we try a ramp tracking sequence for the MT rotator from the MTRotator commander, the rotator moves in the wrong direction, often quite far in the wrong direction. Here is an example, with a very short ramp (so I didn't have to stop it):

            MT Rotator failed ramp 2022-11-03

            {code}
            ramp -25 -26 1
            run_command running command method <bound method RotatorCommander.do_ramp of <lsst.ts.mtrotator.rotator_commander.RotatorCommander object at 0x7fc1390205e0>>
            Tracking a ramp from -25.0 to -26.0 at speed 1.0; this will take 1.00 seconds
            Finished command ramp
            track angle=-25.00 velocity=-1.00
            1667506455.090: controllerState: controllerState=2, offlineSubstate=1, enabledSubstate=2, applicationStatus=5124
            1667506455.090: tracking: tracking=False, lost=True, noNewCommand=False
            1667506455.091: target: position=-25.0000, velocity=-1.0000, tai=1667506455.0936
            1667506455.090: inPosition: inPosition=False
            track angle=-25.15 velocity=-1.00
            1667506455.240: rotation: demandPosition=-25.0000, demandVelocity=-0.0000, demandAcceleration=-0.0600, actualPosition=-25.0000, actualVelocity=-0.0000, debugActualVelocityA=-0.0000, debugActualVelocityB=-0.0000, odometer=392282.8500, timestamp=1667506455.1500
            1667506455.240: tracking: tracking=True, lost=True, noNewCommand=False
            1667506455.240: inPosition: inPosition=True
            1667506455.290: rotation: demandPosition=-25.0000, demandVelocity=-0.0100, demandAcceleration=-0.1200, actualPosition=-25.0000, actualVelocity=-0.0000, debugActualVelocityA=-0.0000, debugActualVelocityB=-0.0000, odometer=392282.8500, timestamp=1667506455.2000
            1667506455.290: target: position=-25.1530, velocity=-1.0000, tai=1667506455.2466
            1667506455.340: rotation: demandPosition=-25.0000, demandVelocity=-0.0100, demandAcceleration=-0.1200, actualPosition=-25.0000, actualVelocity=-0.0200, debugActualVelocityA=-0.0200, debugActualVelocityB=-0.0100, odometer=392282.8500, timestamp=1667506455.2500
            1667506455.340: ccwFollowingError: positionError=-0.0000, velocityError=-0.0200, timestamp=1667506455.2500
            1667506455.390: rotation: demandPosition=-25.0000, demandVelocity=-0.0200, demandAcceleration=-0.1200, actualPosition=-25.0000, actualVelocity=-0.0300, debugActualVelocityA=-0.0300, debugActualVelocityB=-0.0300, odometer=392282.8600, timestamp=1667506455.3000
            1667506455.390: motors: raw=[-2453700000.0, -2451600000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
            1667506455.390: ccwFollowingError: positionError=-0.0000, velocityError=-0.0300, timestamp=1667506455.3000
            track angle=-25.35 velocity=-1.00
            1667506455.440: rotation: demandPosition=-25.0000, demandVelocity=-0.0300, demandAcceleration=-0.1200, actualPosition=-25.0000, actualVelocity=-0.0300, debugActualVelocityA=-0.0300, debugActualVelocityB=-0.0300, odometer=392282.8600, timestamp=1667506455.3500
            1667506455.490: motors: raw=[-2453600000.0, -2451500000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
            1667506455.490: target: position=-25.3514, velocity=-1.0000, tai=1667506455.4450
            1667506455.540: rotation: demandPosition=-25.0100, demandVelocity=-0.0400, demandAcceleration=-0.1200, actualPosition=-25.0100, actualVelocity=-0.0400, debugActualVelocityA=-0.0400, debugActualVelocityB=-0.0400, odometer=392282.8600, timestamp=1667506455.4500
            1667506455.540: motors: raw=[-2453600000.0, -2451400000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
            1667506455.540: ccwFollowingError: positionError=-0.0100, velocityError=-0.0400, timestamp=1667506455.4500
            1667506455.590: motors: raw=[-2453500000.0, -2451400000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
            track angle=-25.55 velocity=-1.00
            1667506455.640: rotation: demandPosition=-25.0100, demandVelocity=-0.0500, demandAcceleration=-0.1200, actualPosition=-25.0100, actualVelocity=-0.0500, debugActualVelocityA=-0.0500, debugActualVelocityB=-0.0500, odometer=392282.8600, timestamp=1667506455.5500
            1667506455.640: motors: raw=[-2453400000.0, -2451300000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
            1667506455.640: ccwFollowingError: positionError=-0.0100, velocityError=-0.0500, timestamp=1667506455.5500
            1667506455.690: rotation: demandPosition=-25.0100, demandVelocity=-0.0600, demandAcceleration=-0.1200, actualPosition=-25.0100, actualVelocity=-0.0600, debugActualVelocityA=-0.0600, debugActualVelocityB=-0.0600, odometer=392282.8700, timestamp=1667506455.6000
            1667506455.690: motors: raw=[-2453300000.0, -2451200000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
            1667506455.690: ccwFollowingError: positionError=-0.0100, velocityError=-0.0600, timestamp=1667506455.6000
            1667506455.690: target: position=-25.5512, velocity=-1.0000, tai=1667506455.6448
            1667506455.740: rotation: demandPosition=-25.0200, demandVelocity=-0.0600, demandAcceleration=-0.1200, actualPosition=-25.0200, actualVelocity=-0.0600, debugActualVelocityA=-0.0600, debugActualVelocityB=-0.0600, odometer=392282.8700, timestamp=1667506455.6500
            1667506455.740: motors: raw=[-2453200000.0, -2451100000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
            1667506455.741: ccwFollowingError: positionError=-0.0200, velocityError=-0.0600, timestamp=1667506455.6500
            1667506455.790: rotation: demandPosition=-25.0200, demandVelocity=-0.0700, demandAcceleration=-0.1200, actualPosition=-25.0200, actualVelocity=-0.0700, debugActualVelocityA=-0.0700, debugActualVelocityB=-0.0700, odometer=392282.8700, timestamp=1667506455.7000
            1667506455.790: motors: raw=[-2453100000.0, -2451000000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
            1667506455.791: ccwFollowingError: positionError=-0.0200, velocityError=-0.0700, timestamp=1667506455.7000
            track angle=-25.75 velocity=-1.00
            1667506455.840: rotation: demandPosition=-25.0200, demandVelocity=-0.0800, demandAcceleration=-0.1200, actualPosition=-25.0200, actualVelocity=-0.0800, debugActualVelocityA=-0.0800, debugActualVelocityB=-0.0800, odometer=392282.8800, timestamp=1667506455.7500
            1667506455.840: motors: raw=[-2452900000.0, -2450800000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
            1667506455.840: ccwFollowingError: positionError=-0.0200, velocityError=-0.0800, timestamp=1667506455.7500
            1667506455.890: rotation: demandPosition=-25.0300, demandVelocity=-0.0800, demandAcceleration=-0.1200, actualPosition=-25.0300, actualVelocity=-0.0800, debugActualVelocityA=-0.0800, debugActualVelocityB=-0.0800, odometer=392282.8800, timestamp=1667506455.8000
            1667506455.890: motors: raw=[-2452800000.0, -2450700000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
            1667506455.890: ccwFollowingError: positionError=-0.0300, velocityError=-0.0800, timestamp=1667506455.8000
            1667506455.890: target: position=-25.7517, velocity=-1.0000, tai=1667506455.8453
            1667506455.940: rotation: demandPosition=-25.0300, demandVelocity=-0.0900, demandAcceleration=-0.1200, actualPosition=-25.0300, actualVelocity=-0.0900, debugActualVelocityA=-0.0900, debugActualVelocityB=-0.0800, odometer=392282.8800, timestamp=1667506455.8500
            1667506455.940: motors: raw=[-2452600000.0, -2450500000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
            1667506455.940: ccwFollowingError: positionError=-0.0300, velocityError=-0.0900, timestamp=1667506455.8500
            1667506455.990: rotation: demandPosition=-25.0400, demandVelocity=-0.0900, demandAcceleration=-0.1200, actualPosition=-25.0400, actualVelocity=-0.0900, debugActualVelocityA=-0.0900, debugActualVelocityB=-0.0900, odometer=392282.8900, timestamp=1667506455.9000
            1667506455.990: motors: raw=[-2452500000.0, -2450300000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
            1667506455.990: ccwFollowingError: positionError=-0.0400, velocityError=-0.0900, timestamp=1667506455.9000
            track angle=-25.95 velocity=-1.00
            1667506456.040: rotation: demandPosition=-25.0400, demandVelocity=-0.1000, demandAcceleration=-0.1100, actualPosition=-25.0400, actualVelocity=-0.1000, debugActualVelocityA=-0.1000, debugActualVelocityB=-0.1000, odometer=392282.8900, timestamp=1667506455.9500
            1667506456.040: motors: raw=[-2452300000.0, -2450200000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
            1667506456.041: ccwFollowingError: positionError=-0.0400, velocityError=-0.1000, timestamp=1667506455.9500
            1667506456.090: rotation: demandPosition=-25.0500, demandVelocity=-0.1000, demandAcceleration=-0.0500, actualPosition=-25.0500, actualVelocity=-0.1000, debugActualVelocityA=-0.1000, debugActualVelocityB=-0.1100, odometer=392282.9000, timestamp=1667506456.0000
            1667506456.090: motors: raw=[-2452100000.0, -2450000000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
            1667506456.090: ccwFollowingError: positionError=-0.0500, velocityError=-0.1100, timestamp=1667506456.0000
            1667506456.091: target: position=-25.9516, velocity=-1.0000, tai=1667506456.0452
            1667506456.140: rotation: demandPosition=-25.0500, demandVelocity=-0.1000, demandAcceleration=0.0200, actualPosition=-25.0500, actualVelocity=-0.1000, debugActualVelocityA=-0.1100, debugActualVelocityB=-0.1000, odometer=392282.9000, timestamp=1667506456.0500
            1667506456.140: motors: raw=[-2451900000.0, -2449800000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
            1667506456.140: ccwFollowingError: positionError=-0.0500, velocityError=-0.1000, timestamp=1667506456.0500
            1667506456.190: rotation: demandPosition=-25.0600, demandVelocity=-0.1000, demandAcceleration=0.1600, actualPosition=-25.0600, actualVelocity=-0.1000, debugActualVelocityA=-0.1000, debugActualVelocityB=-0.1000, odometer=392282.9100, timestamp=1667506456.1000
            1667506456.190: motors: raw=[-2451700000.0, -2449600000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
            track angle=-26.00 velocity=0.00
            1667506456.240: rotation: demandPosition=-25.0600, demandVelocity=-0.0900, demandAcceleration=0.3000, actualPosition=-25.0600, actualVelocity=-0.0900, debugActualVelocityA=-0.0900, debugActualVelocityB=-0.1000, odometer=392282.9100, timestamp=1667506456.1500
            1667506456.240: motors: raw=[-2451500000.0, -2449400000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
            1667506456.240: ccwFollowingError: positionError=-0.0600, velocityError=-0.0900, timestamp=1667506456.1500
            1667506456.290: rotation: demandPosition=-25.0600, demandVelocity=-0.0700, demandAcceleration=0.3700, actualPosition=-25.0600, actualVelocity=-0.0700, debugActualVelocityA=-0.0700, debugActualVelocityB=-0.0700, odometer=392282.9200, timestamp=1667506456.2000
            1667506456.290: motors: raw=[-2451400000.0, -2449200000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
            1667506456.291: ccwFollowingError: positionError=-0.0600, velocityError=-0.0700, timestamp=1667506456.2000
            1667506456.290: tracking: tracking=False, lost=True, noNewCommand=False
            1667506456.291: target: position=-26.0000, velocity=0.0000, tai=1667506456.2452
            1667506456.290: inPosition: inPosition=False
            {code}

            This shows an attempt to track a path from -25 to -26 degrees at 1 deg/second (thus a very short ramp). The rotator starts moving in the correct direction but soon turns around. The tracking commands look like this: {{track angle=-25.95 velocity=-1.00}} (though you can also get the same information from the "target: " events, which the CSC publishes when it sends a track command or point-to-point move command to the low-level controller).

            I also attached a graph from [~julen.garcia] showing position vs time (of both rotator and camera cable wrap). This shows the rotator starting off in the correct direction, but then turning around and ending up in the wrong direction from the starting point. Also notice that the end of the motion is very abrupt; I doubt it is properly jerk-limited.             		In the last two days every time we try a ramp tracking sequence for the MT rotator from the MTRotator commander, the rotator moves in the wrong direction, often quite far in the wrong direction. Here is an example, with a very short ramp (so I didn't have to stop it), but the examples attached are much easier to understand. See the graph

            MT Rotator failed ramp 2022-11-03

            {code}
            ramp -25 -26 1
            run_command running command method <bound method RotatorCommander.do_ramp of <lsst.ts.mtrotator.rotator_commander.RotatorCommander object at 0x7fc1390205e0>>
            Tracking a ramp from -25.0 to -26.0 at speed 1.0; this will take 1.00 seconds
            Finished command ramp
            track angle=-25.00 velocity=-1.00
            1667506455.090: controllerState: controllerState=2, offlineSubstate=1, enabledSubstate=2, applicationStatus=5124
            1667506455.090: tracking: tracking=False, lost=True, noNewCommand=False
            1667506455.091: target: position=-25.0000, velocity=-1.0000, tai=1667506455.0936
            1667506455.090: inPosition: inPosition=False
            track angle=-25.15 velocity=-1.00
            1667506455.240: rotation: demandPosition=-25.0000, demandVelocity=-0.0000, demandAcceleration=-0.0600, actualPosition=-25.0000, actualVelocity=-0.0000, debugActualVelocityA=-0.0000, debugActualVelocityB=-0.0000, odometer=392282.8500, timestamp=1667506455.1500
            1667506455.240: tracking: tracking=True, lost=True, noNewCommand=False
            1667506455.240: inPosition: inPosition=True
            1667506455.290: rotation: demandPosition=-25.0000, demandVelocity=-0.0100, demandAcceleration=-0.1200, actualPosition=-25.0000, actualVelocity=-0.0000, debugActualVelocityA=-0.0000, debugActualVelocityB=-0.0000, odometer=392282.8500, timestamp=1667506455.2000
            1667506455.290: target: position=-25.1530, velocity=-1.0000, tai=1667506455.2466
            1667506455.340: rotation: demandPosition=-25.0000, demandVelocity=-0.0100, demandAcceleration=-0.1200, actualPosition=-25.0000, actualVelocity=-0.0200, debugActualVelocityA=-0.0200, debugActualVelocityB=-0.0100, odometer=392282.8500, timestamp=1667506455.2500
            1667506455.340: ccwFollowingError: positionError=-0.0000, velocityError=-0.0200, timestamp=1667506455.2500
            1667506455.390: rotation: demandPosition=-25.0000, demandVelocity=-0.0200, demandAcceleration=-0.1200, actualPosition=-25.0000, actualVelocity=-0.0300, debugActualVelocityA=-0.0300, debugActualVelocityB=-0.0300, odometer=392282.8600, timestamp=1667506455.3000
            1667506455.390: motors: raw=[-2453700000.0, -2451600000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
            1667506455.390: ccwFollowingError: positionError=-0.0000, velocityError=-0.0300, timestamp=1667506455.3000
            track angle=-25.35 velocity=-1.00
            1667506455.440: rotation: demandPosition=-25.0000, demandVelocity=-0.0300, demandAcceleration=-0.1200, actualPosition=-25.0000, actualVelocity=-0.0300, debugActualVelocityA=-0.0300, debugActualVelocityB=-0.0300, odometer=392282.8600, timestamp=1667506455.3500
            1667506455.490: motors: raw=[-2453600000.0, -2451500000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
            1667506455.490: target: position=-25.3514, velocity=-1.0000, tai=1667506455.4450
            1667506455.540: rotation: demandPosition=-25.0100, demandVelocity=-0.0400, demandAcceleration=-0.1200, actualPosition=-25.0100, actualVelocity=-0.0400, debugActualVelocityA=-0.0400, debugActualVelocityB=-0.0400, odometer=392282.8600, timestamp=1667506455.4500
            1667506455.540: motors: raw=[-2453600000.0, -2451400000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
            1667506455.540: ccwFollowingError: positionError=-0.0100, velocityError=-0.0400, timestamp=1667506455.4500
            1667506455.590: motors: raw=[-2453500000.0, -2451400000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
            track angle=-25.55 velocity=-1.00
            1667506455.640: rotation: demandPosition=-25.0100, demandVelocity=-0.0500, demandAcceleration=-0.1200, actualPosition=-25.0100, actualVelocity=-0.0500, debugActualVelocityA=-0.0500, debugActualVelocityB=-0.0500, odometer=392282.8600, timestamp=1667506455.5500
            1667506455.640: motors: raw=[-2453400000.0, -2451300000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
            1667506455.640: ccwFollowingError: positionError=-0.0100, velocityError=-0.0500, timestamp=1667506455.5500
            1667506455.690: rotation: demandPosition=-25.0100, demandVelocity=-0.0600, demandAcceleration=-0.1200, actualPosition=-25.0100, actualVelocity=-0.0600, debugActualVelocityA=-0.0600, debugActualVelocityB=-0.0600, odometer=392282.8700, timestamp=1667506455.6000
            1667506455.690: motors: raw=[-2453300000.0, -2451200000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
            1667506455.690: ccwFollowingError: positionError=-0.0100, velocityError=-0.0600, timestamp=1667506455.6000
            1667506455.690: target: position=-25.5512, velocity=-1.0000, tai=1667506455.6448
            1667506455.740: rotation: demandPosition=-25.0200, demandVelocity=-0.0600, demandAcceleration=-0.1200, actualPosition=-25.0200, actualVelocity=-0.0600, debugActualVelocityA=-0.0600, debugActualVelocityB=-0.0600, odometer=392282.8700, timestamp=1667506455.6500
            1667506455.740: motors: raw=[-2453200000.0, -2451100000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
            1667506455.741: ccwFollowingError: positionError=-0.0200, velocityError=-0.0600, timestamp=1667506455.6500
            1667506455.790: rotation: demandPosition=-25.0200, demandVelocity=-0.0700, demandAcceleration=-0.1200, actualPosition=-25.0200, actualVelocity=-0.0700, debugActualVelocityA=-0.0700, debugActualVelocityB=-0.0700, odometer=392282.8700, timestamp=1667506455.7000
            1667506455.790: motors: raw=[-2453100000.0, -2451000000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
            1667506455.791: ccwFollowingError: positionError=-0.0200, velocityError=-0.0700, timestamp=1667506455.7000
            track angle=-25.75 velocity=-1.00
            1667506455.840: rotation: demandPosition=-25.0200, demandVelocity=-0.0800, demandAcceleration=-0.1200, actualPosition=-25.0200, actualVelocity=-0.0800, debugActualVelocityA=-0.0800, debugActualVelocityB=-0.0800, odometer=392282.8800, timestamp=1667506455.7500
            1667506455.840: motors: raw=[-2452900000.0, -2450800000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
            1667506455.840: ccwFollowingError: positionError=-0.0200, velocityError=-0.0800, timestamp=1667506455.7500
            1667506455.890: rotation: demandPosition=-25.0300, demandVelocity=-0.0800, demandAcceleration=-0.1200, actualPosition=-25.0300, actualVelocity=-0.0800, debugActualVelocityA=-0.0800, debugActualVelocityB=-0.0800, odometer=392282.8800, timestamp=1667506455.8000
            1667506455.890: motors: raw=[-2452800000.0, -2450700000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
            1667506455.890: ccwFollowingError: positionError=-0.0300, velocityError=-0.0800, timestamp=1667506455.8000
            1667506455.890: target: position=-25.7517, velocity=-1.0000, tai=1667506455.8453
            1667506455.940: rotation: demandPosition=-25.0300, demandVelocity=-0.0900, demandAcceleration=-0.1200, actualPosition=-25.0300, actualVelocity=-0.0900, debugActualVelocityA=-0.0900, debugActualVelocityB=-0.0800, odometer=392282.8800, timestamp=1667506455.8500
            1667506455.940: motors: raw=[-2452600000.0, -2450500000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
            1667506455.940: ccwFollowingError: positionError=-0.0300, velocityError=-0.0900, timestamp=1667506455.8500
            1667506455.990: rotation: demandPosition=-25.0400, demandVelocity=-0.0900, demandAcceleration=-0.1200, actualPosition=-25.0400, actualVelocity=-0.0900, debugActualVelocityA=-0.0900, debugActualVelocityB=-0.0900, odometer=392282.8900, timestamp=1667506455.9000
            1667506455.990: motors: raw=[-2452500000.0, -2450300000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
            1667506455.990: ccwFollowingError: positionError=-0.0400, velocityError=-0.0900, timestamp=1667506455.9000
            track angle=-25.95 velocity=-1.00
            1667506456.040: rotation: demandPosition=-25.0400, demandVelocity=-0.1000, demandAcceleration=-0.1100, actualPosition=-25.0400, actualVelocity=-0.1000, debugActualVelocityA=-0.1000, debugActualVelocityB=-0.1000, odometer=392282.8900, timestamp=1667506455.9500
            1667506456.040: motors: raw=[-2452300000.0, -2450200000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
            1667506456.041: ccwFollowingError: positionError=-0.0400, velocityError=-0.1000, timestamp=1667506455.9500
            1667506456.090: rotation: demandPosition=-25.0500, demandVelocity=-0.1000, demandAcceleration=-0.0500, actualPosition=-25.0500, actualVelocity=-0.1000, debugActualVelocityA=-0.1000, debugActualVelocityB=-0.1100, odometer=392282.9000, timestamp=1667506456.0000
            1667506456.090: motors: raw=[-2452100000.0, -2450000000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
            1667506456.090: ccwFollowingError: positionError=-0.0500, velocityError=-0.1100, timestamp=1667506456.0000
            1667506456.091: target: position=-25.9516, velocity=-1.0000, tai=1667506456.0452
            1667506456.140: rotation: demandPosition=-25.0500, demandVelocity=-0.1000, demandAcceleration=0.0200, actualPosition=-25.0500, actualVelocity=-0.1000, debugActualVelocityA=-0.1100, debugActualVelocityB=-0.1000, odometer=392282.9000, timestamp=1667506456.0500
            1667506456.140: motors: raw=[-2451900000.0, -2449800000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
            1667506456.140: ccwFollowingError: positionError=-0.0500, velocityError=-0.1000, timestamp=1667506456.0500
            1667506456.190: rotation: demandPosition=-25.0600, demandVelocity=-0.1000, demandAcceleration=0.1600, actualPosition=-25.0600, actualVelocity=-0.1000, debugActualVelocityA=-0.1000, debugActualVelocityB=-0.1000, odometer=392282.9100, timestamp=1667506456.1000
            1667506456.190: motors: raw=[-2451700000.0, -2449600000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
            track angle=-26.00 velocity=0.00
            1667506456.240: rotation: demandPosition=-25.0600, demandVelocity=-0.0900, demandAcceleration=0.3000, actualPosition=-25.0600, actualVelocity=-0.0900, debugActualVelocityA=-0.0900, debugActualVelocityB=-0.1000, odometer=392282.9100, timestamp=1667506456.1500
            1667506456.240: motors: raw=[-2451500000.0, -2449400000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
            1667506456.240: ccwFollowingError: positionError=-0.0600, velocityError=-0.0900, timestamp=1667506456.1500
            1667506456.290: rotation: demandPosition=-25.0600, demandVelocity=-0.0700, demandAcceleration=0.3700, actualPosition=-25.0600, actualVelocity=-0.0700, debugActualVelocityA=-0.0700, debugActualVelocityB=-0.0700, odometer=392282.9200, timestamp=1667506456.2000
            1667506456.290: motors: raw=[-2451400000.0, -2449200000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
            1667506456.291: ccwFollowingError: positionError=-0.0600, velocityError=-0.0700, timestamp=1667506456.2000
            1667506456.290: tracking: tracking=False, lost=True, noNewCommand=False
            1667506456.291: target: position=-26.0000, velocity=0.0000, tai=1667506456.2452
            1667506456.290: inPosition: inPosition=False
            {code}

            This shows an attempt to track a path from -25 to -26 degrees at 1 deg/second (thus a very short ramp). The rotator starts moving in the correct direction but soon turns around. The tracking commands look like this: {{track angle=-25.95 velocity=-1.00}} (though you can also get the same information from the "target: " events, which the CSC publishes when it sends a track command or point-to-point move command to the low-level controller).

            I also attached a graph from [~julen.garcia] showing position vs time (of both rotator and camera cable wrap). This shows the rotator starting off in the correct direction, but then turning around and ending up in the wrong direction from the starting point. Also notice that the end of the motion is very abrupt; I doubt it is properly jerk-limited.
            rowen Russell Owen made changes -
            Description In the last two days every time we try a ramp tracking sequence for the MT rotator from the MTRotator commander, the rotator moves in the wrong direction, often quite far in the wrong direction. Here is an example, with a very short ramp (so I didn't have to stop it), but the examples attached are much easier to understand. See the graph

            MT Rotator failed ramp 2022-11-03

            {code}
            ramp -25 -26 1
            run_command running command method <bound method RotatorCommander.do_ramp of <lsst.ts.mtrotator.rotator_commander.RotatorCommander object at 0x7fc1390205e0>>
            Tracking a ramp from -25.0 to -26.0 at speed 1.0; this will take 1.00 seconds
            Finished command ramp
            track angle=-25.00 velocity=-1.00
            1667506455.090: controllerState: controllerState=2, offlineSubstate=1, enabledSubstate=2, applicationStatus=5124
            1667506455.090: tracking: tracking=False, lost=True, noNewCommand=False
            1667506455.091: target: position=-25.0000, velocity=-1.0000, tai=1667506455.0936
            1667506455.090: inPosition: inPosition=False
            track angle=-25.15 velocity=-1.00
            1667506455.240: rotation: demandPosition=-25.0000, demandVelocity=-0.0000, demandAcceleration=-0.0600, actualPosition=-25.0000, actualVelocity=-0.0000, debugActualVelocityA=-0.0000, debugActualVelocityB=-0.0000, odometer=392282.8500, timestamp=1667506455.1500
            1667506455.240: tracking: tracking=True, lost=True, noNewCommand=False
            1667506455.240: inPosition: inPosition=True
            1667506455.290: rotation: demandPosition=-25.0000, demandVelocity=-0.0100, demandAcceleration=-0.1200, actualPosition=-25.0000, actualVelocity=-0.0000, debugActualVelocityA=-0.0000, debugActualVelocityB=-0.0000, odometer=392282.8500, timestamp=1667506455.2000
            1667506455.290: target: position=-25.1530, velocity=-1.0000, tai=1667506455.2466
            1667506455.340: rotation: demandPosition=-25.0000, demandVelocity=-0.0100, demandAcceleration=-0.1200, actualPosition=-25.0000, actualVelocity=-0.0200, debugActualVelocityA=-0.0200, debugActualVelocityB=-0.0100, odometer=392282.8500, timestamp=1667506455.2500
            1667506455.340: ccwFollowingError: positionError=-0.0000, velocityError=-0.0200, timestamp=1667506455.2500
            1667506455.390: rotation: demandPosition=-25.0000, demandVelocity=-0.0200, demandAcceleration=-0.1200, actualPosition=-25.0000, actualVelocity=-0.0300, debugActualVelocityA=-0.0300, debugActualVelocityB=-0.0300, odometer=392282.8600, timestamp=1667506455.3000
            1667506455.390: motors: raw=[-2453700000.0, -2451600000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
            1667506455.390: ccwFollowingError: positionError=-0.0000, velocityError=-0.0300, timestamp=1667506455.3000
            track angle=-25.35 velocity=-1.00
            1667506455.440: rotation: demandPosition=-25.0000, demandVelocity=-0.0300, demandAcceleration=-0.1200, actualPosition=-25.0000, actualVelocity=-0.0300, debugActualVelocityA=-0.0300, debugActualVelocityB=-0.0300, odometer=392282.8600, timestamp=1667506455.3500
            1667506455.490: motors: raw=[-2453600000.0, -2451500000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
            1667506455.490: target: position=-25.3514, velocity=-1.0000, tai=1667506455.4450
            1667506455.540: rotation: demandPosition=-25.0100, demandVelocity=-0.0400, demandAcceleration=-0.1200, actualPosition=-25.0100, actualVelocity=-0.0400, debugActualVelocityA=-0.0400, debugActualVelocityB=-0.0400, odometer=392282.8600, timestamp=1667506455.4500
            1667506455.540: motors: raw=[-2453600000.0, -2451400000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
            1667506455.540: ccwFollowingError: positionError=-0.0100, velocityError=-0.0400, timestamp=1667506455.4500
            1667506455.590: motors: raw=[-2453500000.0, -2451400000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
            track angle=-25.55 velocity=-1.00
            1667506455.640: rotation: demandPosition=-25.0100, demandVelocity=-0.0500, demandAcceleration=-0.1200, actualPosition=-25.0100, actualVelocity=-0.0500, debugActualVelocityA=-0.0500, debugActualVelocityB=-0.0500, odometer=392282.8600, timestamp=1667506455.5500
            1667506455.640: motors: raw=[-2453400000.0, -2451300000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
            1667506455.640: ccwFollowingError: positionError=-0.0100, velocityError=-0.0500, timestamp=1667506455.5500
            1667506455.690: rotation: demandPosition=-25.0100, demandVelocity=-0.0600, demandAcceleration=-0.1200, actualPosition=-25.0100, actualVelocity=-0.0600, debugActualVelocityA=-0.0600, debugActualVelocityB=-0.0600, odometer=392282.8700, timestamp=1667506455.6000
            1667506455.690: motors: raw=[-2453300000.0, -2451200000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
            1667506455.690: ccwFollowingError: positionError=-0.0100, velocityError=-0.0600, timestamp=1667506455.6000
            1667506455.690: target: position=-25.5512, velocity=-1.0000, tai=1667506455.6448
            1667506455.740: rotation: demandPosition=-25.0200, demandVelocity=-0.0600, demandAcceleration=-0.1200, actualPosition=-25.0200, actualVelocity=-0.0600, debugActualVelocityA=-0.0600, debugActualVelocityB=-0.0600, odometer=392282.8700, timestamp=1667506455.6500
            1667506455.740: motors: raw=[-2453200000.0, -2451100000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
            1667506455.741: ccwFollowingError: positionError=-0.0200, velocityError=-0.0600, timestamp=1667506455.6500
            1667506455.790: rotation: demandPosition=-25.0200, demandVelocity=-0.0700, demandAcceleration=-0.1200, actualPosition=-25.0200, actualVelocity=-0.0700, debugActualVelocityA=-0.0700, debugActualVelocityB=-0.0700, odometer=392282.8700, timestamp=1667506455.7000
            1667506455.790: motors: raw=[-2453100000.0, -2451000000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
            1667506455.791: ccwFollowingError: positionError=-0.0200, velocityError=-0.0700, timestamp=1667506455.7000
            track angle=-25.75 velocity=-1.00
            1667506455.840: rotation: demandPosition=-25.0200, demandVelocity=-0.0800, demandAcceleration=-0.1200, actualPosition=-25.0200, actualVelocity=-0.0800, debugActualVelocityA=-0.0800, debugActualVelocityB=-0.0800, odometer=392282.8800, timestamp=1667506455.7500
            1667506455.840: motors: raw=[-2452900000.0, -2450800000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
            1667506455.840: ccwFollowingError: positionError=-0.0200, velocityError=-0.0800, timestamp=1667506455.7500
            1667506455.890: rotation: demandPosition=-25.0300, demandVelocity=-0.0800, demandAcceleration=-0.1200, actualPosition=-25.0300, actualVelocity=-0.0800, debugActualVelocityA=-0.0800, debugActualVelocityB=-0.0800, odometer=392282.8800, timestamp=1667506455.8000
            1667506455.890: motors: raw=[-2452800000.0, -2450700000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
            1667506455.890: ccwFollowingError: positionError=-0.0300, velocityError=-0.0800, timestamp=1667506455.8000
            1667506455.890: target: position=-25.7517, velocity=-1.0000, tai=1667506455.8453
            1667506455.940: rotation: demandPosition=-25.0300, demandVelocity=-0.0900, demandAcceleration=-0.1200, actualPosition=-25.0300, actualVelocity=-0.0900, debugActualVelocityA=-0.0900, debugActualVelocityB=-0.0800, odometer=392282.8800, timestamp=1667506455.8500
            1667506455.940: motors: raw=[-2452600000.0, -2450500000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
            1667506455.940: ccwFollowingError: positionError=-0.0300, velocityError=-0.0900, timestamp=1667506455.8500
            1667506455.990: rotation: demandPosition=-25.0400, demandVelocity=-0.0900, demandAcceleration=-0.1200, actualPosition=-25.0400, actualVelocity=-0.0900, debugActualVelocityA=-0.0900, debugActualVelocityB=-0.0900, odometer=392282.8900, timestamp=1667506455.9000
            1667506455.990: motors: raw=[-2452500000.0, -2450300000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
            1667506455.990: ccwFollowingError: positionError=-0.0400, velocityError=-0.0900, timestamp=1667506455.9000
            track angle=-25.95 velocity=-1.00
            1667506456.040: rotation: demandPosition=-25.0400, demandVelocity=-0.1000, demandAcceleration=-0.1100, actualPosition=-25.0400, actualVelocity=-0.1000, debugActualVelocityA=-0.1000, debugActualVelocityB=-0.1000, odometer=392282.8900, timestamp=1667506455.9500
            1667506456.040: motors: raw=[-2452300000.0, -2450200000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
            1667506456.041: ccwFollowingError: positionError=-0.0400, velocityError=-0.1000, timestamp=1667506455.9500
            1667506456.090: rotation: demandPosition=-25.0500, demandVelocity=-0.1000, demandAcceleration=-0.0500, actualPosition=-25.0500, actualVelocity=-0.1000, debugActualVelocityA=-0.1000, debugActualVelocityB=-0.1100, odometer=392282.9000, timestamp=1667506456.0000
            1667506456.090: motors: raw=[-2452100000.0, -2450000000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
            1667506456.090: ccwFollowingError: positionError=-0.0500, velocityError=-0.1100, timestamp=1667506456.0000
            1667506456.091: target: position=-25.9516, velocity=-1.0000, tai=1667506456.0452
            1667506456.140: rotation: demandPosition=-25.0500, demandVelocity=-0.1000, demandAcceleration=0.0200, actualPosition=-25.0500, actualVelocity=-0.1000, debugActualVelocityA=-0.1100, debugActualVelocityB=-0.1000, odometer=392282.9000, timestamp=1667506456.0500
            1667506456.140: motors: raw=[-2451900000.0, -2449800000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
            1667506456.140: ccwFollowingError: positionError=-0.0500, velocityError=-0.1000, timestamp=1667506456.0500
            1667506456.190: rotation: demandPosition=-25.0600, demandVelocity=-0.1000, demandAcceleration=0.1600, actualPosition=-25.0600, actualVelocity=-0.1000, debugActualVelocityA=-0.1000, debugActualVelocityB=-0.1000, odometer=392282.9100, timestamp=1667506456.1000
            1667506456.190: motors: raw=[-2451700000.0, -2449600000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
            track angle=-26.00 velocity=0.00
            1667506456.240: rotation: demandPosition=-25.0600, demandVelocity=-0.0900, demandAcceleration=0.3000, actualPosition=-25.0600, actualVelocity=-0.0900, debugActualVelocityA=-0.0900, debugActualVelocityB=-0.1000, odometer=392282.9100, timestamp=1667506456.1500
            1667506456.240: motors: raw=[-2451500000.0, -2449400000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
            1667506456.240: ccwFollowingError: positionError=-0.0600, velocityError=-0.0900, timestamp=1667506456.1500
            1667506456.290: rotation: demandPosition=-25.0600, demandVelocity=-0.0700, demandAcceleration=0.3700, actualPosition=-25.0600, actualVelocity=-0.0700, debugActualVelocityA=-0.0700, debugActualVelocityB=-0.0700, odometer=392282.9200, timestamp=1667506456.2000
            1667506456.290: motors: raw=[-2451400000.0, -2449200000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
            1667506456.291: ccwFollowingError: positionError=-0.0600, velocityError=-0.0700, timestamp=1667506456.2000
            1667506456.290: tracking: tracking=False, lost=True, noNewCommand=False
            1667506456.291: target: position=-26.0000, velocity=0.0000, tai=1667506456.2452
            1667506456.290: inPosition: inPosition=False
            {code}

            This shows an attempt to track a path from -25 to -26 degrees at 1 deg/second (thus a very short ramp). The rotator starts moving in the correct direction but soon turns around. The tracking commands look like this: {{track angle=-25.95 velocity=-1.00}} (though you can also get the same information from the "target: " events, which the CSC publishes when it sends a track command or point-to-point move command to the low-level controller).

            I also attached a graph from [~julen.garcia] showing position vs time (of both rotator and camera cable wrap). This shows the rotator starting off in the correct direction, but then turning around and ending up in the wrong direction from the starting point. Also notice that the end of the motion is very abrupt; I doubt it is properly jerk-limited.             		In the last two days every time we try a ramp tracking sequence for the MT rotator from the MTRotator commander, the rotator moves in the wrong direction, often quite far in the wrong direction. Here is an example, with a very short ramp (so I didn't have to stop it):

            MT Rotator failed ramp 2022-11-03

            {code}
            ramp -25 -26 1
            run_command running command method <bound method RotatorCommander.do_ramp of <lsst.ts.mtrotator.rotator_commander.RotatorCommander object at 0x7fc1390205e0>>
            Tracking a ramp from -25.0 to -26.0 at speed 1.0; this will take 1.00 seconds
            Finished command ramp
            track angle=-25.00 velocity=-1.00
            1667506455.090: controllerState: controllerState=2, offlineSubstate=1, enabledSubstate=2, applicationStatus=5124
            1667506455.090: tracking: tracking=False, lost=True, noNewCommand=False
            1667506455.091: target: position=-25.0000, velocity=-1.0000, tai=1667506455.0936
            1667506455.090: inPosition: inPosition=False
            track angle=-25.15 velocity=-1.00
            1667506455.240: rotation: demandPosition=-25.0000, demandVelocity=-0.0000, demandAcceleration=-0.0600, actualPosition=-25.0000, actualVelocity=-0.0000, debugActualVelocityA=-0.0000, debugActualVelocityB=-0.0000, odometer=392282.8500, timestamp=1667506455.1500
            1667506455.240: tracking: tracking=True, lost=True, noNewCommand=False
            1667506455.240: inPosition: inPosition=True
            1667506455.290: rotation: demandPosition=-25.0000, demandVelocity=-0.0100, demandAcceleration=-0.1200, actualPosition=-25.0000, actualVelocity=-0.0000, debugActualVelocityA=-0.0000, debugActualVelocityB=-0.0000, odometer=392282.8500, timestamp=1667506455.2000
            1667506455.290: target: position=-25.1530, velocity=-1.0000, tai=1667506455.2466
            1667506455.340: rotation: demandPosition=-25.0000, demandVelocity=-0.0100, demandAcceleration=-0.1200, actualPosition=-25.0000, actualVelocity=-0.0200, debugActualVelocityA=-0.0200, debugActualVelocityB=-0.0100, odometer=392282.8500, timestamp=1667506455.2500
            1667506455.340: ccwFollowingError: positionError=-0.0000, velocityError=-0.0200, timestamp=1667506455.2500
            1667506455.390: rotation: demandPosition=-25.0000, demandVelocity=-0.0200, demandAcceleration=-0.1200, actualPosition=-25.0000, actualVelocity=-0.0300, debugActualVelocityA=-0.0300, debugActualVelocityB=-0.0300, odometer=392282.8600, timestamp=1667506455.3000
            1667506455.390: motors: raw=[-2453700000.0, -2451600000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
            1667506455.390: ccwFollowingError: positionError=-0.0000, velocityError=-0.0300, timestamp=1667506455.3000
            track angle=-25.35 velocity=-1.00
            1667506455.440: rotation: demandPosition=-25.0000, demandVelocity=-0.0300, demandAcceleration=-0.1200, actualPosition=-25.0000, actualVelocity=-0.0300, debugActualVelocityA=-0.0300, debugActualVelocityB=-0.0300, odometer=392282.8600, timestamp=1667506455.3500
            1667506455.490: motors: raw=[-2453600000.0, -2451500000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
            1667506455.490: target: position=-25.3514, velocity=-1.0000, tai=1667506455.4450
            1667506455.540: rotation: demandPosition=-25.0100, demandVelocity=-0.0400, demandAcceleration=-0.1200, actualPosition=-25.0100, actualVelocity=-0.0400, debugActualVelocityA=-0.0400, debugActualVelocityB=-0.0400, odometer=392282.8600, timestamp=1667506455.4500
            1667506455.540: motors: raw=[-2453600000.0, -2451400000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
            1667506455.540: ccwFollowingError: positionError=-0.0100, velocityError=-0.0400, timestamp=1667506455.4500
            1667506455.590: motors: raw=[-2453500000.0, -2451400000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
            track angle=-25.55 velocity=-1.00
            1667506455.640: rotation: demandPosition=-25.0100, demandVelocity=-0.0500, demandAcceleration=-0.1200, actualPosition=-25.0100, actualVelocity=-0.0500, debugActualVelocityA=-0.0500, debugActualVelocityB=-0.0500, odometer=392282.8600, timestamp=1667506455.5500
            1667506455.640: motors: raw=[-2453400000.0, -2451300000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
            1667506455.640: ccwFollowingError: positionError=-0.0100, velocityError=-0.0500, timestamp=1667506455.5500
            1667506455.690: rotation: demandPosition=-25.0100, demandVelocity=-0.0600, demandAcceleration=-0.1200, actualPosition=-25.0100, actualVelocity=-0.0600, debugActualVelocityA=-0.0600, debugActualVelocityB=-0.0600, odometer=392282.8700, timestamp=1667506455.6000
            1667506455.690: motors: raw=[-2453300000.0, -2451200000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
            1667506455.690: ccwFollowingError: positionError=-0.0100, velocityError=-0.0600, timestamp=1667506455.6000
            1667506455.690: target: position=-25.5512, velocity=-1.0000, tai=1667506455.6448
            1667506455.740: rotation: demandPosition=-25.0200, demandVelocity=-0.0600, demandAcceleration=-0.1200, actualPosition=-25.0200, actualVelocity=-0.0600, debugActualVelocityA=-0.0600, debugActualVelocityB=-0.0600, odometer=392282.8700, timestamp=1667506455.6500
            1667506455.740: motors: raw=[-2453200000.0, -2451100000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
            1667506455.741: ccwFollowingError: positionError=-0.0200, velocityError=-0.0600, timestamp=1667506455.6500
            1667506455.790: rotation: demandPosition=-25.0200, demandVelocity=-0.0700, demandAcceleration=-0.1200, actualPosition=-25.0200, actualVelocity=-0.0700, debugActualVelocityA=-0.0700, debugActualVelocityB=-0.0700, odometer=392282.8700, timestamp=1667506455.7000
            1667506455.790: motors: raw=[-2453100000.0, -2451000000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
            1667506455.791: ccwFollowingError: positionError=-0.0200, velocityError=-0.0700, timestamp=1667506455.7000
            track angle=-25.75 velocity=-1.00
            1667506455.840: rotation: demandPosition=-25.0200, demandVelocity=-0.0800, demandAcceleration=-0.1200, actualPosition=-25.0200, actualVelocity=-0.0800, debugActualVelocityA=-0.0800, debugActualVelocityB=-0.0800, odometer=392282.8800, timestamp=1667506455.7500
            1667506455.840: motors: raw=[-2452900000.0, -2450800000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
            1667506455.840: ccwFollowingError: positionError=-0.0200, velocityError=-0.0800, timestamp=1667506455.7500
            1667506455.890: rotation: demandPosition=-25.0300, demandVelocity=-0.0800, demandAcceleration=-0.1200, actualPosition=-25.0300, actualVelocity=-0.0800, debugActualVelocityA=-0.0800, debugActualVelocityB=-0.0800, odometer=392282.8800, timestamp=1667506455.8000
            1667506455.890: motors: raw=[-2452800000.0, -2450700000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
            1667506455.890: ccwFollowingError: positionError=-0.0300, velocityError=-0.0800, timestamp=1667506455.8000
            1667506455.890: target: position=-25.7517, velocity=-1.0000, tai=1667506455.8453
            1667506455.940: rotation: demandPosition=-25.0300, demandVelocity=-0.0900, demandAcceleration=-0.1200, actualPosition=-25.0300, actualVelocity=-0.0900, debugActualVelocityA=-0.0900, debugActualVelocityB=-0.0800, odometer=392282.8800, timestamp=1667506455.8500
            1667506455.940: motors: raw=[-2452600000.0, -2450500000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
            1667506455.940: ccwFollowingError: positionError=-0.0300, velocityError=-0.0900, timestamp=1667506455.8500
            1667506455.990: rotation: demandPosition=-25.0400, demandVelocity=-0.0900, demandAcceleration=-0.1200, actualPosition=-25.0400, actualVelocity=-0.0900, debugActualVelocityA=-0.0900, debugActualVelocityB=-0.0900, odometer=392282.8900, timestamp=1667506455.9000
            1667506455.990: motors: raw=[-2452500000.0, -2450300000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
            1667506455.990: ccwFollowingError: positionError=-0.0400, velocityError=-0.0900, timestamp=1667506455.9000
            track angle=-25.95 velocity=-1.00
            1667506456.040: rotation: demandPosition=-25.0400, demandVelocity=-0.1000, demandAcceleration=-0.1100, actualPosition=-25.0400, actualVelocity=-0.1000, debugActualVelocityA=-0.1000, debugActualVelocityB=-0.1000, odometer=392282.8900, timestamp=1667506455.9500
            1667506456.040: motors: raw=[-2452300000.0, -2450200000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
            1667506456.041: ccwFollowingError: positionError=-0.0400, velocityError=-0.1000, timestamp=1667506455.9500
            1667506456.090: rotation: demandPosition=-25.0500, demandVelocity=-0.1000, demandAcceleration=-0.0500, actualPosition=-25.0500, actualVelocity=-0.1000, debugActualVelocityA=-0.1000, debugActualVelocityB=-0.1100, odometer=392282.9000, timestamp=1667506456.0000
            1667506456.090: motors: raw=[-2452100000.0, -2450000000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
            1667506456.090: ccwFollowingError: positionError=-0.0500, velocityError=-0.1100, timestamp=1667506456.0000
            1667506456.091: target: position=-25.9516, velocity=-1.0000, tai=1667506456.0452
            1667506456.140: rotation: demandPosition=-25.0500, demandVelocity=-0.1000, demandAcceleration=0.0200, actualPosition=-25.0500, actualVelocity=-0.1000, debugActualVelocityA=-0.1100, debugActualVelocityB=-0.1000, odometer=392282.9000, timestamp=1667506456.0500
            1667506456.140: motors: raw=[-2451900000.0, -2449800000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
            1667506456.140: ccwFollowingError: positionError=-0.0500, velocityError=-0.1000, timestamp=1667506456.0500
            1667506456.190: rotation: demandPosition=-25.0600, demandVelocity=-0.1000, demandAcceleration=0.1600, actualPosition=-25.0600, actualVelocity=-0.1000, debugActualVelocityA=-0.1000, debugActualVelocityB=-0.1000, odometer=392282.9100, timestamp=1667506456.1000
            1667506456.190: motors: raw=[-2451700000.0, -2449600000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
            track angle=-26.00 velocity=0.00
            1667506456.240: rotation: demandPosition=-25.0600, demandVelocity=-0.0900, demandAcceleration=0.3000, actualPosition=-25.0600, actualVelocity=-0.0900, debugActualVelocityA=-0.0900, debugActualVelocityB=-0.1000, odometer=392282.9100, timestamp=1667506456.1500
            1667506456.240: motors: raw=[-2451500000.0, -2449400000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
            1667506456.240: ccwFollowingError: positionError=-0.0600, velocityError=-0.0900, timestamp=1667506456.1500
            1667506456.290: rotation: demandPosition=-25.0600, demandVelocity=-0.0700, demandAcceleration=0.3700, actualPosition=-25.0600, actualVelocity=-0.0700, debugActualVelocityA=-0.0700, debugActualVelocityB=-0.0700, odometer=392282.9200, timestamp=1667506456.2000
            1667506456.290: motors: raw=[-2451400000.0, -2449200000.0], torque=[0.0, 0.0], current=[0.0, 0.0], busVoltage=0.0000
            1667506456.291: ccwFollowingError: positionError=-0.0600, velocityError=-0.0700, timestamp=1667506456.2000
            1667506456.290: tracking: tracking=False, lost=True, noNewCommand=False
            1667506456.291: target: position=-26.0000, velocity=0.0000, tai=1667506456.2452
            1667506456.290: inPosition: inPosition=False
            {code}

            This shows an attempt to track a path from -25 to -26 degrees at 1 deg/second (thus a very short ramp). The rotator starts moving in the correct direction but soon turns around. The tracking commands look like this: {{track angle=-25.95 velocity=-1.00}} (though you can also get the same information from the "target: " events, which the CSC publishes when it sends a track command or point-to-point move command to the low-level controller).

            I also attached a graph from [~julen.garcia] showing position vs time (of both rotator and camera cable wrap). This shows the rotator starting off in the correct direction, but then turning around and ending up in the wrong direction from the starting point. Also notice that the end of the motion is very abrupt; I doubt it is properly jerk-limited.
            Hide
            Permalink
            ttsai Te-Wei Tsai added a comment -

            I think the TMA had the similar issue in the preparation of sock test and it was fixed already. Since the rotator and TMA are using the same trajectory generator code, this should not be a problem anymore (as long as the rotator is back and I deploy the controller with the updated trajectory generator).

            Show
            ttsai Te-Wei Tsai added a comment - I think the TMA had the similar issue in the preparation of sock test and it was fixed already. Since the rotator and TMA are using the same trajectory generator code, this should not be a problem anymore (as long as the rotator is back and I deploy the controller with the updated trajectory generator).
            ttsai Te-Wei Tsai made changes -
            Sprint TSSW Sprint - Dec 19 - Jan 02 [ 1210 ]
            ttsai Te-Wei Tsai made changes -
            Status To Do [ 10001 ] In Progress [ 3 ]
            ttsai Te-Wei Tsai made changes -
            Story Points 0
            Hide
            Permalink
            ttsai Te-Wei Tsai added a comment -

            Please help to review the PR:
            https://github.com/lsst-ts/ts_rotator_controller/pull/47

            Thanks!

            Show
            ttsai Te-Wei Tsai added a comment - Please help to review the PR: https://github.com/lsst-ts/ts_rotator_controller/pull/47 Thanks!
            ttsai Te-Wei Tsai made changes -
            Reviewers Russell Owen [ rowen ]
            Status In Progress [ 3 ] In Review [ 10004 ]
            Hide
            Permalink
            rowen Russell Owen added a comment -

            I would not like close this until we can test it. The code from Tekniker that runs on the rotator is similar to the code in the TMA, but not identical.

            Show
            rowen Russell Owen added a comment - I would not like close this until we can test it. The code from Tekniker that runs on the rotator is similar to the code in the TMA, but not identical.
            Hide
            Permalink
            ttsai Te-Wei Tsai added a comment -

            In this case, I would prefer:
            1. You still review that PR because I would like to put the latest Simulink update by Tekniker (which is the same as TMA base on Ismael) to the develop branch. Although I could create another ticket for this, I prefer not.
            2. I put this ticket back to "To Do" since I do not know when the rotator will be recovered for us to test.

            Please help to comment. Thanks!

            Show
            ttsai Te-Wei Tsai added a comment - In this case, I would prefer: 1. You still review that PR because I would like to put the latest Simulink update by Tekniker (which is the same as TMA base on Ismael) to the develop branch. Although I could create another ticket for this, I prefer not. 2. I put this ticket back to "To Do" since I do not know when the rotator will be recovered for us to test. Please help to comment. Thanks!
            Hide
            Permalink
            rowen Russell Owen added a comment -

            Maybe it makes the most sense to close this one and start a new one if we still have problems. I'll review the PR next week when I'm back at work.

            Show
            rowen Russell Owen added a comment - Maybe it makes the most sense to close this one and start a new one if we still have problems. I'll review the PR next week when I'm back at work.
            aclements Andy Clements made changes -
            Sprint TSSW Sprint - Dec 19 - Jan 02 [ 1210 ] TSSW Sprint - Dec 19 - Jan 02, TSSW Sprint - Jan 02 - Jan 16 [ 1210, 1211 ]
            Hide
            Permalink
            ttsai Te-Wei Tsai added a comment -

            Thanks Russell to review and approve the PR!

            Show
            ttsai Te-Wei Tsai added a comment - Thanks Russell to review and approve the PR!
            ttsai Te-Wei Tsai made changes -
            Status In Review [ 10004 ] Reviewed [ 10101 ]
            ttsai Te-Wei Tsai made changes -
            Resolution Done [ 10000 ]
            Status Reviewed [ 10101 ] Done [ 10002 ]

              People

              Assignee:
              ttsai Te-Wei Tsai
              Reporter:
              rowen Russell Owen
              Reviewers:
              Russell Owen
              Watchers:
              Brian Stalder, Bruno Quint, Julen Garcia, Russell Owen, Te-Wei Tsai
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Jenkins

                  No builds found.