Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-35452

MTM2 stuck in the ENABLED state

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: ts_main_telescope
    • Labels:
    • Story Points:
      0
    • Epic Link:
    • Sprint:
      TSSW Sprint - Jul 04 - Jul 18
    • Team:
      Telescope and Site
    • Urgent?:
      No

      Description

      When running a test on TTS, I tried to send the M2 from the ENABLED state to the STANDBY state. The command failed. I talked to Tiago Ribeiro who confirmed that M2 is stuck in the ENABLED state because it cannot connect to the M2 controller. Here is a print-screen of the command using Jupyter Notebook:

      I will add the full traceback from the CSC as a comment later. 

      Killing the CSC using ArgoCD was the temporary solution. 

        Attachments

          Issue Links

          relates to

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-35454 Update the Conda Recipe of ts_m2

          • Done

            Activity

            Ascending order - Click to sort in descending order
            bquint Bruno Quint created issue -
            bquint Bruno Quint made changes -
            Field Original Value New Value
            Comment [ Here is the full traceback: 
            {code:java}
            Traceback (most recent call last): File "/opt/lsst/software/stack/miniconda/lib/python3.10/site-packages/lsst/ts/m2/csc.py", line 623, in _transition_controller_state await self.model.write_command_to_server( File "/opt/lsst/software/stack/miniconda/lib/python3.10/site-packages/lsst/ts/m2/model.py", line 392, in write_command_to_server raise OSError("No TCP/IP connection.") OSError: No TCP/IP connection. During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/opt/lsst/software/stack/miniconda/lib/python3.10/site-packages/lsst/ts/salobj/topics/controller_command.py", line 233, in _run_callback ack = await result # type: ignore File "/opt/lsst/software/stack/miniconda/lib/python3.10/site-packages/lsst/ts/m2/csc.py", line 572, in do_disable await self._transition_controller_state( File "/opt/lsst/software/stack/miniconda/lib/python3.10/site-packages/lsst/ts/m2/csc.py", line 630, in _transition_controller_state await self.model.close() File "/opt/lsst/software/stack/miniconda/lib/python3.10/site-packages/lsst/ts/m2/model.py", line 322, in close await self.client_telemetry.close() File "/opt/lsst/software/stack/miniconda/lib/python3.10/site-packages/lsst/ts/m2/tcp_client.py", line 398, in close await self._basic_close() File "/opt/lsst/software/stack/miniconda/lib/python3.10/site-packages/lsst/ts/m2/tcp_client.py", line 251, in _basic_close await tcpip.close_stream_writer(self.writer) File "/opt/lsst/software/stack/miniconda/lib/python3.10/site-packages/lsst/ts/tcpip/utils.py", line 54, in close_stream_writer await asyncio.wait_for(writer.wait_closed(), timeout=5) File "/opt/lsst/software/stack/miniconda/lib/python3.10/asyncio/tasks.py", line 445, in wait_for return fut.result() File "/opt/lsst/software/stack/miniconda/lib/python3.10/asyncio/streams.py", line 344, in wait_closed await self._protocol._get_close_waiter(self) File "/opt/lsst/software/stack/miniconda/lib/python3.10/site-packages/lsst/ts/salobj/topics/controller_command.py", line 233, in _run_callback ack = await result # type: ignore File "/opt/lsst/software/stack/miniconda/lib/python3.10/site-packages/lsst/ts/m2/csc.py", line 572, in do_disable await self._transition_controller_state( File "/opt/lsst/software/stack/miniconda/lib/python3.10/site-packages/lsst/ts/m2/csc.py", line 630, in _transition_controller_state await self.model.close() File "/opt/lsst/software/stack/miniconda/lib/python3.10/site-packages/lsst/ts/m2/model.py", line 322, in close await self.client_telemetry.close() File "/opt/lsst/software/stack/miniconda/lib/python3.10/site-packages/lsst/ts/m2/tcp_client.py", line 398, in close await self._basic_close() File "/opt/lsst/software/stack/miniconda/lib/python3.10/site-packages/lsst/ts/m2/tcp_client.py", line 251, in _basic_close await tcpip.close_stream_writer(self.writer) File "/opt/lsst/software/stack/miniconda/lib/python3.10/site-packages/lsst/ts/tcpip/utils.py", line 54, in close_stream_writer await asyncio.wait_for(writer.wait_closed(), timeout=5) File "/opt/lsst/software/stack/miniconda/lib/python3.10/asyncio/tasks.py", line 445, in wait_for return fut.result() File "/opt/lsst/software/stack/miniconda/lib/python3.10/asyncio/streams.py", line 344, in wait_closed await self._protocol._get_close_waiter(self) File "/opt/lsst/software/stack/miniconda/lib/python3.10/site-packages/lsst/ts/salobj/topics/controller_command.py", line 233, in _run_callback ack = await result # type: ignore File "/opt/lsst/software/stack/miniconda/lib/python3.10/site-packages/lsst/ts/m2/csc.py", line 572, in do_disable await self._transition_controller_state( File "/opt/lsst/software/stack/miniconda/lib/python3.10/site-packages/lsst/ts/m2/csc.py", line 630, in _transition_controller_state await self.model.close() File "/opt/lsst/software/stack/miniconda/lib/python3.10/site-packages/lsst/ts/m2/model.py", line 322, in close await self.client_telemetry.close() File "/opt/lsst/software/stack/miniconda/lib/python3.10/site-packages/lsst/ts/m2/tcp_client.py", line 398, in close await self._basic_close() File "/opt/lsst/software/stack/miniconda/lib/python3.10/site-packages/lsst/ts/m2/tcp_client.py", line 251, in _basic_close await tcpip.close_stream_writer(self.writer) File "/opt/lsst/software/stack/miniconda/lib/python3.10/site-packages/lsst/ts/tcpip/utils.py", line 54, in close_stream_writer await asyncio.wait_for(writer.wait_closed(), timeout=5) File "/opt/lsst/software/stack/miniconda/lib/python3.10/asyncio/tasks.py", line 445, in wait_for return fut.result() File "/opt/lsst/software/stack/miniconda/lib/python3.10/asyncio/streams.py", line 344, in wait_closed await self._protocol._get_close_waiter(self) File "/opt/lsst/software/stack/miniconda/lib/python3.10/site-packages/lsst/ts/salobj/topics/controller_command.py", line 233, in _run_callback ack = await result # type: ignore File "/opt/lsst/software/stack/miniconda/lib/python3.10/site-packages/lsst/ts/m2/csc.py", line 572, in do_disable await self._transition_controller_state( File "/opt/lsst/software/stack/miniconda/lib/python3.10/site-packages/lsst/ts/m2/csc.py", line 630, in _transition_controller_state await self.model.close() File "/opt/lsst/software/stack/miniconda/lib/python3.10/site-packages/lsst/ts/m2/model.py", line 322, in close await self.client_telemetry.close() File "/opt/lsst/software/stack/miniconda/lib/python3.10/site-packages/lsst/ts/m2/tcp_client.py", line 398, in close await self._basic_close() File "/opt/lsst/software/stack/miniconda/lib/python3.10/site-packages/lsst/ts/m2/tcp_client.py", line 251, in _basic_close await tcpip.close_stream_writer(self.writer) File "/opt/lsst/software/stack/miniconda/lib/python3.10/site-packages/lsst/ts/tcpip/utils.py", line 54, in close_stream_writer await asyncio.wait_for(writer.wait_closed(), timeout=5) File "/opt/lsst/software/stack/miniconda/lib/python3.10/asyncio/tasks.py", line 445, in wait_for return fut.result() File "/opt/lsst/software/stack/miniconda/lib/python3.10/asyncio/streams.py", line 344, in wait_closed await self._protocol._get_close_waiter(self) File "/opt/lsst/software/stack/miniconda/lib/python3.10/site-packages/lsst/ts/salobj/topics/controller_command.py", line 233, in _run_callback ack = await result # type: ignore File "/opt/lsst/software/stack/miniconda/lib/python3.10/site-packages/lsst/ts/m2/csc.py", line 696, in do_resetForceOffsets await self._write_command_to_server( File "/opt/lsst/software/stack/miniconda/lib/python3.10/site-packages/lsst/ts/m2/csc.py", line 835, in _write_command_to_server await self.model.close() File "/opt/lsst/software/stack/miniconda/lib/python3.10/site-packages/lsst/ts/m2/model.py", line 322, in close await self.client_telemetry.close() File "/opt/lsst/software/stack/miniconda/lib/python3.10/site-packages/lsst/ts/m2/tcp_client.py", line 398, in close await self._basic_close() File "/opt/lsst/software/stack/miniconda/lib/python3.10/site-packages/lsst/ts/m2/tcp_client.py", line 251, in _basic_close await tcpip.close_stream_writer(self.writer) File "/opt/lsst/software/stack/miniconda/lib/python3.10/site-packages/lsst/ts/tcpip/utils.py", line 54, in close_stream_writer await asyncio.wait_for(writer.wait_closed(), timeout=5) File "/opt/lsst/software/stack/miniconda/lib/python3.10/asyncio/tasks.py", line 445, in wait_for return fut.result() File "/opt/lsst/software/stack/miniconda/lib/python3.10/asyncio/streams.py", line 344, in wait_closed await self._protocol._get_close_waiter(self) File "/opt/lsst/software/stack/miniconda/lib/python3.10/site-packages/lsst/ts/salobj/topics/controller_command.py", line 233, in _run_callback ack = await result # type: ignore File "/opt/lsst/software/stack/miniconda/lib/python3.10/site-packages/lsst/ts/m2/csc.py", line 572, in do_disable await self._transition_controller_state( File "/opt/lsst/software/stack/miniconda/lib/python3.10/site-packages/lsst/ts/m2/csc.py", line 630, in _transition_controller_state await self.model.close() File "/opt/lsst/software/stack/miniconda/lib/python3.10/site-packages/lsst/ts/m2/model.py", line 322, in close await self.client_telemetry.close() File "/opt/lsst/software/stack/miniconda/lib/python3.10/site-packages/lsst/ts/m2/tcp_client.py", line 398, in close await self._basic_close() File "/opt/lsst/software/stack/miniconda/lib/python3.10/site-packages/lsst/ts/m2/tcp_client.py", line 251, in _basic_close await tcpip.close_stream_writer(self.writer) File "/opt/lsst/software/stack/miniconda/lib/python3.10/site-packages/lsst/ts/tcpip/utils.py", line 54, in close_stream_writer await asyncio.wait_for(writer.wait_closed(), timeout=5) File "/opt/lsst/software/stack/miniconda/lib/python3.10/asyncio/tasks.py", line 445, in wait_for return fut.result() File "/opt/lsst/software/stack/miniconda/lib/python3.10/asyncio/streams.py", line 344, in wait_closed await self._protocol._get_close_waiter(self) File "/opt/lsst/software/stack/miniconda/lib/python3.10/site-packages/lsst/ts/m2/csc.py", line 429, in _connection_monitor_loop await self.model.close() File "/opt/lsst/software/stack/miniconda/lib/python3.10/site-packages/lsst/ts/m2/model.py", line 322, in close await self.client_telemetry.close() File "/opt/lsst/software/stack/miniconda/lib/python3.10/site-packages/lsst/ts/m2/tcp_client.py", line 398, in close await self._basic_close() File "/opt/lsst/software/stack/miniconda/lib/python3.10/site-packages/lsst/ts/m2/tcp_client.py", line 251, in _basic_close await tcpip.close_stream_writer(self.writer) File "/opt/lsst/software/stack/miniconda/lib/python3.10/site-packages/lsst/ts/tcpip/utils.py", line 54, in close_stream_writer await asyncio.wait_for(writer.wait_closed(), timeout=5) File "/opt/lsst/software/stack/miniconda/lib/python3.10/asyncio/tasks.py", line 445, in wait_for return fut.result() File "/opt/lsst/software/stack/miniconda/lib/python3.10/asyncio/streams.py", line 344, in wait_closed await self._protocol._get_close_waiter(self) File "/opt/lsst/software/stack/miniconda/lib/python3.10/asyncio/selector_events.py", line 854, in _read_ready__data_received data = self._sock.recv(self.max_size) TimeoutError: [Errno 110] Connection timed out
            {code} ]
            ttsai Te-Wei Tsai made changes -
            Assignee Te-Wei Tsai [ ttsai ] Bruno Quint [ bquint ]
            ttsai Te-Wei Tsai made changes -
            Attachment image.png [ 61314 ]
            ttsai Te-Wei Tsai made changes -
            Attachment image.png [ 61314 ]
            ttsai Te-Wei Tsai made changes -
            Link This issue relates to DM-35454 [ DM-35454 ]
            ttsai Te-Wei Tsai made changes -
            Assignee Bruno Quint [ bquint ] Te-Wei Tsai [ ttsai ]
            ttsai Te-Wei Tsai made changes -
            Component/s ts_main_telescope [ 16710 ]
            Epic Link DM-27719 [ 442124 ]
            Sprint TSSW Sprint - Jul 04 - Jul 18 [ 1177 ]
            Labels M2
            ttsai Te-Wei Tsai made changes -
            Status To Do [ 10001 ] In Progress [ 3 ]
            ttsai Te-Wei Tsai made changes -
            Story Points 0
            ttsai Te-Wei Tsai made changes -
            Reviewers Tiago Ribeiro [ tribeiro ]
            Status In Progress [ 3 ] In Review [ 10004 ]
            ttsai Te-Wei Tsai made changes -
            Status In Review [ 10004 ] Reviewed [ 10101 ]
            ttsai Te-Wei Tsai made changes -
            Resolution Done [ 10000 ]
            Status Reviewed [ 10101 ] Done [ 10002 ]

              People

              Assignee:
              ttsai Te-Wei Tsai
              Reporter:
              bquint Bruno Quint
              Reviewers:
              Tiago Ribeiro
              Watchers:
              Bruno Quint, Te-Wei Tsai, Tiago Ribeiro
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Jenkins

                  No builds found.