Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-33806

Hexapod and Rotator communication issue

    XMLWordPrintable

    Details

    • Team:
      Telescope and Site
    • Urgent?:
      No

      Description

      These are the error messages for the Rotator from LOVE:

      beg_enable failed; remaining in state <State.DISABLED: 1>: SET_STATE command with param1=<SetStateParam.START: 1> failed: TimeoutError()
      		 
      2022/02/24 14:09:24 TAI
      		 
      beg_enable failed; remaining in state <State.DISABLED: 1>: SET_STATE command with param1=<SetStateParam.START: 1> failed: TimeoutError()
      		 
      2022/02/24 14:08:49 TAI
      		 
      beg_enable failed; remaining in state <State.DISABLED: 1>: SET_STATE command with param1=<SetStateParam.START: 1> failed: TimeoutError()
      		 
      2022/02/24 14:08:37 TAI
      		 
      beg_enable failed; remaining in state <State.DISABLED: 1>: SET_STATE command with param1=<SetStateParam.START: 1> failed: TimeoutError()
      		 
      2022/02/24 14:08:19 TAI
      		 
      beg_enable failed; remaining in state <State.DISABLED: 1>: SET_STATE command with param1=<SetStateParam.START: 1> failed: TimeoutError()
      		 
      2022/02/24 14:08:14 TAI
      		 
      connect to host=rot-pxi-controller.cp.lsst.org, port=5570
      		 
      2022/02/24 14:07:30 TAI
      		 
      Fault! errorCode=1, errorReport=‘Low-level controller went to FAULT state’
      		 
      2022/02/24 14:07:26 TAI
      		 
      Camera cable wrap not following closely enough: error # 8 = 2.2182482744325043 > 2.2 deg
      		 
      2022/02/24 14:07:26 TAI
      		 
      Camera cable wrap not following closely enough: error # 7 = 2.227866255594215 > 2.2 deg
      		 
      2022/02/24 14:07:26 TAI
      		 
      Camera cable wrap not following closely enough: error # 6 = 2.238405505656445 > 2.2 deg
      		 
      2022/02/24 14:07:26 TAI
      		 
      Camera cable wrap not following closely enough: error # 5 = 2.229053719183888 > 2.2 deg
      		 
      2022/02/24 14:07:26 TAI
      		 
      Camera cable wrap not following closely enough: error # 4 = 2.2306347156732826 > 2.2 deg
      		 
      2022/02/24 14:07:26 TAI
      		 
      Camera cable wrap not following closely enough: error # 3 = 2.2198391775972084 > 2.2 deg
      		 
      2022/02/24 13:58:53 TAI
      		 
      beg_enable failed; remaining in state <State.DISABLED: 1>: SET_STATE command with param1=<SetStateParam.START: 1> failed: ExpectedError(‘Not in Standby state’)
      		 
      2022/02/24 13:58:53 TAI
      		 
      connect to host=rot-pxi-controller.cp.lsst.org, port=5570
      		 
      2022/02/24 13:58:40 TAI
      		 
      beg_enable failed; remaining in state <State.DISABLED: 1>: SET_STATE command with param1=<SetStateParam.START: 1> failed: TimeoutError()
      		 
      2022/02/24 13:58:35 TAI
      		 
      connect to host=rot-pxi-controller.cp.lsst.org, port=5570
      		 
      2022/02/24 13:58:28 TAI
      		 
      beg_enable failed; remaining in state <State.DISABLED: 1>: SET_STATE command with param1=<SetStateParam.START: 1> failed: TimeoutError()
      		 
      2022/02/24 13:58:10 TAI
      		 
      beg_enable failed; remaining in state <State.DISABLED: 1>: SET_STATE command with param1=<SetStateParam.START: 1> failed: TimeoutError()
      		 
      2022/02/24 13:57:49 TAI
      		 
      beg_enable failed; remaining in state <State.DISABLED: 1>: SET_STATE command with param1=<SetStateParam.START: 1> failed: TimeoutError()
      		 
      2022/02/24 13:57:44 TAI
      		 
      connect to host=rot-pxi-controller.cp.lsst.org, port=5570
      		 
      2022/02/24 13:56:37 TAI
      		 
      Fault! errorCode=1, errorReport=‘Low-level controller went to FAULT state’
      		 
      2022/02/24 13:56:34 TAI
      		 
      Camera cable wrap not following closely enough: error # 3 = -2.2011053263890403 > 2.2 deg
      		 
      2022/02/24 13:44:15 TAI
      		 
      beg_enable failed; remaining in state <State.DISABLED: 1>: SET_STATE command with param1=<SetStateParam.START: 1> failed: ExpectedError(‘Not in Standby state’)
      		 
      2022/02/24 13:44:15 TAI
      		 
      connect to host=rot-pxi-controller.cp.lsst.org, port=5570
      		 
      2022/02/24 13:43:57 TAI
      		 
      beg_enable failed; remaining in state <State.DISABLED: 1>: SET_STATE command with param1=<SetStateParam.START: 1> failed: TimeoutError()
      		 
      2022/02/24 13:43:45 TAI
      		 
      beg_enable failed; remaining in state <State.DISABLED: 1>: SET_STATE command with param1=<SetStateParam.START: 1> failed: TimeoutError()
      		 
      2022/02/24 13:43:39 TAI
      		 
      connect to host=rot-pxi-controller.cp.lsst.org, port=5570
      		 
      2022/02/24 13:43:04 TAI
      		 
      Fault! errorCode=1, errorReport=‘Low-level controller went to FAULT state’
      		 
      2022/02/24 13:43:01 TAI
      		 
      Camera cable wrap not following closely enough: error # 9 = 2.219396143715284 > 2.2 deg
      		 
      2022/02/24 13:43:01 TAI
      		 
      Camera cable wrap not following closely enough: error # 8 = 2.256213968612692 > 2.2 deg
      		 
      2022/02/24 13:43:01 TAI
      		 
      Camera cable wrap not following closely enough: error # 7 = 2.3053848717087027 > 2.2 deg
      		 
      2022/02/24 13:43:01 TAI
      		 
      Camera cable wrap not following closely enough: error # 6 = 2.3067554532498233 > 2.2 deg
      		 
      2022/02/24 13:43:01 TAI
      		 
      Camera cable wrap not following closely enough: error # 5 = 2.316286488987476 > 2.2 deg
      		 
      2022/02/24 13:43:01 TAI
      		 
      Camera cable wrap not following closely enough: error # 4 = 2.528497855527351 > 2.2 deg
      		 
      2022/02/24 13:43:01 TAI
      		 
      Camera cable wrap not following closely enough: error # 3 = 2.255121035307493 > 2.2 deg
      		 
      2022/02/24 13:32:44 TAI
      		 
      connect to host=rot-pxi-controller.cp.lsst.org, port=5570
      		 
      2022/02/24 13:31:22 TAI
      		 
      Fault! errorCode=1, errorReport=‘Low-level controller went to FAULT state’
      		 
      2022/02/24 13:31:19 TAI
      		 
      Camera cable wrap not following closely enough: error # 3 = -2.2032868004283728 > 2.2 deg
      		 
      2022/02/24 13:18:19 TAI
      		 
      connect to host=rot-pxi-controller.cp.lsst.org, port=5570
      		 
      2022/02/24 13:15:32 TAI
      		 
      Fault! errorCode=1, errorReport=‘Low-level controller went to FAULT state’
      		 
      2022/02/24 13:15:29 TAI
      		 
      Camera cable wrap not following closely enough: error # 5 = 2.2193617400468213 > 2.2 deg
      		 
      2022/02/24 13:15:29 TAI
      		 
      Camera cable wrap not following closely enough: error # 4 = 2.2523672215932535 > 2.2 deg
      		 
      2022/02/24 13:15:28 TAI
      		 
      Camera cable wrap not following closely enough: error # 3 = 2.267643136110294 > 2.2 deg
      		 
      2022/02/24 13:15:28 TAI
      		 
      Camera cable wrap not following closely enough: error # 5 = 2.2634913206963247 > 2.2 deg
      		 
      2022/02/24 13:15:28 TAI
      		 
      Camera cable wrap not following closely enough: error # 4 = 2.250567835639039 > 2.2 deg
      		 
      2022/02/24 13:15:28 TAI
      		 
      Camera cable wrap not following closely enough: error # 3 = 2.2302412941895042 > 2.2 deg
      		 
      2022/02/24 13:06:58 TAI
      		 
      beg_enable failed; remaining in state <State.DISABLED: 1>: SET_STATE command with param1=<SetStateParam.START: 1> failed: TimeoutError()

       

      Here are the fault for the hexapod:

      2022/02/24 15:03:08 TAI
      		 
      Fault! errorCode=2, errorReport=‘Timed out connecting to host=10.9.57.226, port=5560’
      		 
      2022/02/24 15:02:58 TAI
      		 
      connect to host=10.9.57.226, port=5560
      		 
      2022/02/24 15:02:51 TAI
      		 
      Fault! errorCode=2, errorReport=‘Timed out connecting to host=10.9.57.226, port=5560’
      		 
      2022/02/24 15:02:41 TAI
      		 
      connect to host=10.9.57.226, port=5560
      		 
      2022/02/24 15:02:23 TAI
      		 
      Fault! errorCode=2, errorReport=‘Timed out connecting to host=10.9.57.226, port=5560’
      		 
      2022/02/24 15:02:13 TAI
      		 
      connect to host=10.9.57.226, port=5560
      		 
      2022/02/24 15:00:34 TAI
      		 
      Fault! errorCode=2, errorReport=‘Timed out connecting to host=10.9.57.226, port=5560’
      		 
      2022/02/24 15:00:24 TAI
      		 
      connect to host=10.9.57.226, port=5560
      		 
      2022/02/24 14:30:16 TAI
      		 
      Fault! errorCode=1, errorReport=‘Low-level controller went to FAULT state’
      		 
      2022/02/24 14:21:26 TAI
      		 
      max_move_duration=61.74
      		 
      2022/02/24 14:21:25 TAI
      		 
      connect to host=camhex-pxi-controller.cp.lsst.org, port=5560
      		 
      2022/02/23 20:22:41 TAI
      		 
      Disabling the CSC because the low-level controller is in non-enabled state 1.0

      Sending command needed to be several times to change the state of the state machine. Especially when trying to clear faults.  Communication related?

        Attachments

          Activity

          Ascending order - Click to sort in descending order
          4 older comments
          Hide
          Permalink
          hdrass Holger Drass added a comment -

          I would describe it as barely, I had to send command several times to change the state of the state machine. Especially when trying to clear faults. I can not be sure but I guess this is also because of the communication. Updating the description.

          Show
          hdrass Holger Drass added a comment - I would describe it as barely, I had to send command several times to change the state of the state machine. Especially when trying to clear faults. I can not be sure but I guess this is also because of the communication. Updating the description.
          Hide
          Permalink
          ttsai Te-Wei Tsai added a comment - - edited

          For the rotator messages, the bug should be fixed in DM-32902. The related one is DM-33774.

          For the hexapod messages, please check why we have the 10.9.57.226. The IP of Camera Hexapod PXI is camhex-pxi-controller.cp.lsst.org or 139.229.178.8.

          The following is the neighboring IP from hexrot server:

          (base) [ttsai@hexrot ~]$ ip neigh
          		 
          139.229.178.254 dev em1 lladdr 00:22:bd:f8:19:ff REACHABLE
          		 
          10.9.57.229 dev em2  FAILED
          		 
          10.9.57.226 dev em2  FAILED
          		 
          139.229.178.5 dev em1 lladdr 00:80:2f:23:7e:be STALE
          		 
          139.229.178.8 dev em1 lladdr 00:80:2f:21:da:c1 STALE

          You can see 10.9.57.226 is dead.

          Show
          ttsai Te-Wei Tsai added a comment - - edited For the rotator messages, the bug should be fixed in DM-32902 . The related one is DM-33774 . For the hexapod messages, please check why we have the 10.9.57.226 . The IP of Camera Hexapod PXI is camhex-pxi-controller.cp.lsst.org or 139.229.178.8 . The following is the neighboring IP from hexrot server: (base) [ttsai@hexrot ~]$ ip neigh 139.229.178.254 dev em1 lladdr 00:22:bd:f8:19:ff REACHABLE 10.9.57.229 dev em2 FAILED 10.9.57.226 dev em2 FAILED 139.229.178.5 dev em1 lladdr 00:80:2f:23:7e:be STALE 139.229.178.8 dev em1 lladdr 00:80:2f:21:da:c1 STALE You can see 10.9.57.226 is dead.
          Hide
          Permalink
          bquint Bruno Quint added a comment -

          I could enable MTHexapod_1 by running the following command without the settings:

          await mtcs.set_state(state=salobj.State.ENABLED, components=["mthexapod_1"] )

           

          Show
          bquint Bruno Quint added a comment - I could enable MTHexapod_1 by running the following command without the settings: await mtcs.set_state(state=salobj.State.ENABLED, components=[ "mthexapod_1" ] )  
          Hide
          Permalink
          hdrass Holger Drass added a comment -

          The hexapod problem seems to be solved. I did not appear again during the last testing sessions.

          The rotator issue is still existing. It seems not to be a communication issue but more of a missing feature on the CCW to allow a point-to-point movement. It is not a blocker for the ticket but make extents the time needed to execute the test case by the amount of time needed to recover from the faults. Closing this ticket.

          Show
          hdrass Holger Drass added a comment - The hexapod problem seems to be solved. I did not appear again during the last testing sessions. The rotator issue is still existing. It seems not to be a communication issue but more of a missing feature on the CCW to allow a point-to-point movement. It is not a blocker for the ticket but make extents the time needed to execute the test case by the amount of time needed to recover from the faults. Closing this ticket.
          Hide
          Permalink
          hdrass Holger Drass added a comment -

          This issue was solved here: https://jira.lsstcorp.org/browse/SUMMIT-6237

          Closing this ticket.

          Show
          hdrass Holger Drass added a comment - This issue was solved here: https://jira.lsstcorp.org/browse/SUMMIT-6237 Closing this ticket.

            People

            Assignee:
            hdrass Holger Drass
            Reporter:
            hdrass Holger Drass
            Watchers:
            Andy Clements, Bruno Quint, Holger Drass, HyeYun Park, Te-Wei Tsai
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved:

                Jenkins

                No builds found.