Traceback (most recent call last): File "/opt/lsst/software/stack/miniconda/lib/python3.10/site-packages/lsst/ts/m2/csc.py", line 623, in transition_controller_state await self.model.write_command_to_server( File "/opt/lsst/software/stack/miniconda/lib/python3.10/site-packages/lsst/ts/m2/model.py", line 392, in write_command_to_server raise OSError("No TCP/IP connection.") OSError: No TCP/IP connection. During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/opt/lsst/software/stack/miniconda/lib/python3.10/site-packages/lsst/ts/salobj/topics/controller_command.py", line 233, in _run_callback ack = await result # type: ignore File "/opt/lsst/software/stack/miniconda/lib/python3.10/site-packages/lsst/ts/m2/csc.py", line 572, in do_disable await self._transition_controller_state( File "/opt/lsst/software/stack/miniconda/lib/python3.10/site-packages/lsst/ts/m2/csc.py", line 630, in _transition_controller_state await self.model.close() File "/opt/lsst/software/stack/miniconda/lib/python3.10/site-packages/lsst/ts/m2/model.py", line 322, in close await self.client_telemetry.close() File "/opt/lsst/software/stack/miniconda/lib/python3.10/site-packages/lsst/ts/m2/tcp_client.py", line 398, in close await self._basic_close() File "/opt/lsst/software/stack/miniconda/lib/python3.10/site-packages/lsst/ts/m2/tcp_client.py", line 251, in _basic_close await tcpip.close_stream_writer(self.writer) File "/opt/lsst/software/stack/miniconda/lib/python3.10/site-packages/lsst/ts/tcpip/utils.py", line 54, in close_stream_writer await asyncio.wait_for(writer.wait_closed(), timeout=5) File "/opt/lsst/software/stack/miniconda/lib/python3.10/asyncio/tasks.py", line 445, in wait_for return fut.result() File "/opt/lsst/software/stack/miniconda/lib/python3.10/asyncio/streams.py", line 344, in wait_closed await self._protocol._get_close_waiter(self) File "/opt/lsst/software/stack/miniconda/lib/python3.10/site-packages/lsst/ts/salobj/topics/controller_command.py", line 233, in _run_callback ack = await result # type: ignore File "/opt/lsst/software/stack/miniconda/lib/python3.10/site-packages/lsst/ts/m2/csc.py", line 572, in do_disable await self._transition_controller_state( File "/opt/lsst/software/stack/miniconda/lib/python3.10/site-packages/lsst/ts/m2/csc.py", line 630, in _transition_controller_state await self.model.close() File "/opt/lsst/software/stack/miniconda/lib/python3.10/site-packages/lsst/ts/m2/model.py", line 322, in close await self.client_telemetry.close() File "/opt/lsst/software/stack/miniconda/lib/python3.10/site-packages/lsst/ts/m2/tcp_client.py", line 398, in close await self._basic_close() File "/opt/lsst/software/stack/miniconda/lib/python3.10/site-packages/lsst/ts/m2/tcp_client.py", line 251, in _basic_close await tcpip.close_stream_writer(self.writer) File "/opt/lsst/software/stack/miniconda/lib/python3.10/site-packages/lsst/ts/tcpip/utils.py", line 54, in close_stream_writer await asyncio.wait_for(writer.wait_closed(), timeout=5) File "/opt/lsst/software/stack/miniconda/lib/python3.10/asyncio/tasks.py", line 445, in wait_for return fut.result() File "/opt/lsst/software/stack/miniconda/lib/python3.10/asyncio/streams.py", line 344, in wait_closed await self._protocol._get_close_waiter(self) File "/opt/lsst/software/stack/miniconda/lib/python3.10/site-packages/lsst/ts/salobj/topics/controller_command.py", line 233, in _run_callback ack = await result # type: ignore File "/opt/lsst/software/stack/miniconda/lib/python3.10/site-packages/lsst/ts/m2/csc.py", line 572, in do_disable await self._transition_controller_state( File "/opt/lsst/software/stack/miniconda/lib/python3.10/site-packages/lsst/ts/m2/csc.py", line 630, in _transition_controller_state await self.model.close() File "/opt/lsst/software/stack/miniconda/lib/python3.10/site-packages/lsst/ts/m2/model.py", line 322, in close await self.client_telemetry.close() File "/opt/lsst/software/stack/miniconda/lib/python3.10/site-packages/lsst/ts/m2/tcp_client.py", line 398, in close await self._basic_close() File "/opt/lsst/software/stack/miniconda/lib/python3.10/site-packages/lsst/ts/m2/tcp_client.py", line 251, in _basic_close await tcpip.close_stream_writer(self.writer) File "/opt/lsst/software/stack/miniconda/lib/python3.10/site-packages/lsst/ts/tcpip/utils.py", line 54, in close_stream_writer await asyncio.wait_for(writer.wait_closed(), timeout=5) File "/opt/lsst/software/stack/miniconda/lib/python3.10/asyncio/tasks.py", line 445, in wait_for return fut.result() File "/opt/lsst/software/stack/miniconda/lib/python3.10/asyncio/streams.py", line 344, in wait_closed await self._protocol._get_close_waiter(self) File "/opt/lsst/software/stack/miniconda/lib/python3.10/site-packages/lsst/ts/salobj/topics/controller_command.py", line 233, in _run_callback ack = await result # type: ignore File "/opt/lsst/software/stack/miniconda/lib/python3.10/site-packages/lsst/ts/m2/csc.py", line 572, in do_disable await self._transition_controller_state( File "/opt/lsst/software/stack/miniconda/lib/python3.10/site-packages/lsst/ts/m2/csc.py", line 630, in _transition_controller_state await self.model.close() File "/opt/lsst/software/stack/miniconda/lib/python3.10/site-packages/lsst/ts/m2/model.py", line 322, in close await self.client_telemetry.close() File "/opt/lsst/software/stack/miniconda/lib/python3.10/site-packages/lsst/ts/m2/tcp_client.py", line 398, in close await self._basic_close() File "/opt/lsst/software/stack/miniconda/lib/python3.10/site-packages/lsst/ts/m2/tcp_client.py", line 251, in _basic_close await tcpip.close_stream_writer(self.writer) File "/opt/lsst/software/stack/miniconda/lib/python3.10/site-packages/lsst/ts/tcpip/utils.py", line 54, in close_stream_writer await asyncio.wait_for(writer.wait_closed(), timeout=5) File "/opt/lsst/software/stack/miniconda/lib/python3.10/asyncio/tasks.py", line 445, in wait_for return fut.result() File "/opt/lsst/software/stack/miniconda/lib/python3.10/asyncio/streams.py", line 344, in wait_closed await self._protocol._get_close_waiter(self) File "/opt/lsst/software/stack/miniconda/lib/python3.10/site-packages/lsst/ts/salobj/topics/controller_command.py", line 233, in _run_callback ack = await result # type: ignore File "/opt/lsst/software/stack/miniconda/lib/python3.10/site-packages/lsst/ts/m2/csc.py", line 696, in do_resetForceOffsets await self._write_command_to_server( File "/opt/lsst/software/stack/miniconda/lib/python3.10/site-packages/lsst/ts/m2/csc.py", line 835, in _write_command_to_server await self.model.close() File "/opt/lsst/software/stack/miniconda/lib/python3.10/site-packages/lsst/ts/m2/model.py", line 322, in close await self.client_telemetry.close() File "/opt/lsst/software/stack/miniconda/lib/python3.10/site-packages/lsst/ts/m2/tcp_client.py", line 398, in close await self._basic_close() File "/opt/lsst/software/stack/miniconda/lib/python3.10/site-packages/lsst/ts/m2/tcp_client.py", line 251, in _basic_close await tcpip.close_stream_writer(self.writer) File "/opt/lsst/software/stack/miniconda/lib/python3.10/site-packages/lsst/ts/tcpip/utils.py", line 54, in close_stream_writer await asyncio.wait_for(writer.wait_closed(), timeout=5) File "/opt/lsst/software/stack/miniconda/lib/python3.10/asyncio/tasks.py", line 445, in wait_for return fut.result() File "/opt/lsst/software/stack/miniconda/lib/python3.10/asyncio/streams.py", line 344, in wait_closed await self._protocol._get_close_waiter(self) File "/opt/lsst/software/stack/miniconda/lib/python3.10/site-packages/lsst/ts/salobj/topics/controller_command.py", line 233, in _run_callback ack = await result # type: ignore File "/opt/lsst/software/stack/miniconda/lib/python3.10/site-packages/lsst/ts/m2/csc.py", line 572, in do_disable await self._transition_controller_state( File "/opt/lsst/software/stack/miniconda/lib/python3.10/site-packages/lsst/ts/m2/csc.py", line 630, in _transition_controller_state await self.model.close() File "/opt/lsst/software/stack/miniconda/lib/python3.10/site-packages/lsst/ts/m2/model.py", line 322, in close await self.client_telemetry.close() File "/opt/lsst/software/stack/miniconda/lib/python3.10/site-packages/lsst/ts/m2/tcp_client.py", line 398, in close await self._basic_close() File "/opt/lsst/software/stack/miniconda/lib/python3.10/site-packages/lsst/ts/m2/tcp_client.py", line 251, in _basic_close await tcpip.close_stream_writer(self.writer) File "/opt/lsst/software/stack/miniconda/lib/python3.10/site-packages/lsst/ts/tcpip/utils.py", line 54, in close_stream_writer await asyncio.wait_for(writer.wait_closed(), timeout=5) File "/opt/lsst/software/stack/miniconda/lib/python3.10/asyncio/tasks.py", line 445, in wait_for return fut.result() File "/opt/lsst/software/stack/miniconda/lib/python3.10/asyncio/streams.py", line 344, in wait_closed await self._protocol._get_close_waiter(self) File "/opt/lsst/software/stack/miniconda/lib/python3.10/site-packages/lsst/ts/m2/csc.py", line 429, in _connection_monitor_loop await self.model.close() File "/opt/lsst/software/stack/miniconda/lib/python3.10/site-packages/lsst/ts/m2/model.py", line 322, in close await self.client_telemetry.close() File "/opt/lsst/software/stack/miniconda/lib/python3.10/site-packages/lsst/ts/m2/tcp_client.py", line 398, in close await self._basic_close() File "/opt/lsst/software/stack/miniconda/lib/python3.10/site-packages/lsst/ts/m2/tcp_client.py", line 251, in _basic_close await tcpip.close_stream_writer(self.writer) File "/opt/lsst/software/stack/miniconda/lib/python3.10/site-packages/lsst/ts/tcpip/utils.py", line 54, in close_stream_writer await asyncio.wait_for(writer.wait_closed(), timeout=5) File "/opt/lsst/software/stack/miniconda/lib/python3.10/asyncio/tasks.py", line 445, in wait_for return fut.result() File "/opt/lsst/software/stack/miniconda/lib/python3.10/asyncio/streams.py", line 344, in wait_closed await self._protocol._get_close_waiter(self) File "/opt/lsst/software/stack/miniconda/lib/python3.10/asyncio/selector_events.py", line 854, in _read_ready_data_received data = self._sock.recv(self.max_size) TimeoutError: [Errno 110] Connection timed out

Bruno Quint Please try the ts_m2 instead if you think it is from M2 CSC. I do not know the mtcs at all. Thanks! 

Based on the error message, you can see the connection timeout. Therefore, you need to check  what is the host name and ports of M2 at this moment. I think it should be reported in the log.

Te-Wei Tsai, please, look at the traceback Bruno sent on the message above... you can ignore the MTCS error.

The traceback above comes from the CSC itself. What I think we need to do is add "try/except" clauses around potential failures and ignore them when doing the transition from ENABLED to DISABLED, and DISABLED to STANDBY, while doing what is needed to cleanup any pending connection.

In the case above, I think we want to protect this call from failure, as you can see from:

 await self._transition_controller_state( File "/opt/lsst/software/stack/miniconda/lib/python3.10/site-packages/lsst/ts/m2/csc.py", line 630

This is interesting. I am not sure this is right or not. The error message is connection timeout. Therefore, I am guessing the hostname or ports are not right. They are configured here:

https://github.com/lsst-ts/ts_config_mttcs/tree/develop/MTM2/v2

There is the log message here to check:
https://github.com/lsst-ts/ts_m2/blob/ef85c2f2413a4479db97931ae10be389771795e6/python/lsst/ts/m2/csc.py#L486-L488

Then, if the server closed the connection, the CSC will transition to the Fault state actually:
https://github.com/lsst-ts/ts_m2/blob/ef85c2f2413a4479db97931ae10be389771795e6/python/lsst/ts/m2/csc.py#L408-L416

In addition, if for some how the above function does not work, there should be no telemetry at that time. And there is the related warning message here:

https://github.com/lsst-ts/ts_m2/blob/ef85c2f2413a4479db97931ae10be389771795e6/python/lsst/ts/m2/csc.py#L365-L371

Therefore, it is weird to me actually.

I think it is better to check the log message first to see any clue to know what happened at that time.

I will debug this at DM-35454 to use the story point together.

Bruno Quint Please help to provide the related timestamp for me to query the log messages at the EFD. Thanks! This is at the Tucson Teststand instead of summit. Right?

Some similar discussion here:
https://github.com/home-assistant/core/issues/10468

Please help to review the PR:
https://github.com/lsst-ts/ts_m2/pull/27

Thanks!

This should be resolved in DM-35454.