Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-35452

MTM2 stuck in the ENABLED state

    XMLWordPrintable

Details

    • Bug
    • Status: Done
    • Resolution: Done
    • None
    • ts_main_telescope
    • 0
    • TSSW Sprint - Jul 04 - Jul 18
    • Telescope and Site
    • No

    Description

      When running a test on TTS, I tried to send the M2 from the ENABLED state to the STANDBY state. The command failed. I talked to tribeiro who confirmed that M2 is stuck in the ENABLED state because it cannot connect to the M2 controller. Here is a print-screen of the command using Jupyter Notebook:

      I will add the full traceback from the CSC as a comment later. 

      Killing the CSC using ArgoCD was the temporary solution. 

      Attachments

        Issue Links

          Activity

            bquint Bruno Quint added a comment - - edited

            Traceback (most recent call last): File "/opt/lsst/software/stack/miniconda/lib/python3.10/site-packages/lsst/ts/m2/csc.py", line 623, in transition_controller_state await self.model.write_command_to_server( File "/opt/lsst/software/stack/miniconda/lib/python3.10/site-packages/lsst/ts/m2/model.py", line 392, in write_command_to_server raise OSError("No TCP/IP connection.") OSError: No TCP/IP connection. During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/opt/lsst/software/stack/miniconda/lib/python3.10/site-packages/lsst/ts/salobj/topics/controller_command.py", line 233, in _run_callback ack = await result # type: ignore File "/opt/lsst/software/stack/miniconda/lib/python3.10/site-packages/lsst/ts/m2/csc.py", line 572, in do_disable await self._transition_controller_state( File "/opt/lsst/software/stack/miniconda/lib/python3.10/site-packages/lsst/ts/m2/csc.py", line 630, in _transition_controller_state await self.model.close() File "/opt/lsst/software/stack/miniconda/lib/python3.10/site-packages/lsst/ts/m2/model.py", line 322, in close await self.client_telemetry.close() File "/opt/lsst/software/stack/miniconda/lib/python3.10/site-packages/lsst/ts/m2/tcp_client.py", line 398, in close await self._basic_close() File "/opt/lsst/software/stack/miniconda/lib/python3.10/site-packages/lsst/ts/m2/tcp_client.py", line 251, in _basic_close await tcpip.close_stream_writer(self.writer) File "/opt/lsst/software/stack/miniconda/lib/python3.10/site-packages/lsst/ts/tcpip/utils.py", line 54, in close_stream_writer await asyncio.wait_for(writer.wait_closed(), timeout=5) File "/opt/lsst/software/stack/miniconda/lib/python3.10/asyncio/tasks.py", line 445, in wait_for return fut.result() File "/opt/lsst/software/stack/miniconda/lib/python3.10/asyncio/streams.py", line 344, in wait_closed await self._protocol._get_close_waiter(self) File "/opt/lsst/software/stack/miniconda/lib/python3.10/site-packages/lsst/ts/salobj/topics/controller_command.py", line 233, in _run_callback ack = await result # type: ignore File "/opt/lsst/software/stack/miniconda/lib/python3.10/site-packages/lsst/ts/m2/csc.py", line 572, in do_disable await self._transition_controller_state( File "/opt/lsst/software/stack/miniconda/lib/python3.10/site-packages/lsst/ts/m2/csc.py", line 630, in _transition_controller_state await self.model.close() File "/opt/lsst/software/stack/miniconda/lib/python3.10/site-packages/lsst/ts/m2/model.py", line 322, in close await self.client_telemetry.close() File "/opt/lsst/software/stack/miniconda/lib/python3.10/site-packages/lsst/ts/m2/tcp_client.py", line 398, in close await self._basic_close() File "/opt/lsst/software/stack/miniconda/lib/python3.10/site-packages/lsst/ts/m2/tcp_client.py", line 251, in _basic_close await tcpip.close_stream_writer(self.writer) File "/opt/lsst/software/stack/miniconda/lib/python3.10/site-packages/lsst/ts/tcpip/utils.py", line 54, in close_stream_writer await asyncio.wait_for(writer.wait_closed(), timeout=5) File "/opt/lsst/software/stack/miniconda/lib/python3.10/asyncio/tasks.py", line 445, in wait_for return fut.result() File "/opt/lsst/software/stack/miniconda/lib/python3.10/asyncio/streams.py", line 344, in wait_closed await self._protocol._get_close_waiter(self) File "/opt/lsst/software/stack/miniconda/lib/python3.10/site-packages/lsst/ts/salobj/topics/controller_command.py", line 233, in _run_callback ack = await result # type: ignore File "/opt/lsst/software/stack/miniconda/lib/python3.10/site-packages/lsst/ts/m2/csc.py", line 572, in do_disable await self._transition_controller_state( File "/opt/lsst/software/stack/miniconda/lib/python3.10/site-packages/lsst/ts/m2/csc.py", line 630, in _transition_controller_state await self.model.close() File "/opt/lsst/software/stack/miniconda/lib/python3.10/site-packages/lsst/ts/m2/model.py", line 322, in close await self.client_telemetry.close() File "/opt/lsst/software/stack/miniconda/lib/python3.10/site-packages/lsst/ts/m2/tcp_client.py", line 398, in close await self._basic_close() File "/opt/lsst/software/stack/miniconda/lib/python3.10/site-packages/lsst/ts/m2/tcp_client.py", line 251, in _basic_close await tcpip.close_stream_writer(self.writer) File "/opt/lsst/software/stack/miniconda/lib/python3.10/site-packages/lsst/ts/tcpip/utils.py", line 54, in close_stream_writer await asyncio.wait_for(writer.wait_closed(), timeout=5) File "/opt/lsst/software/stack/miniconda/lib/python3.10/asyncio/tasks.py", line 445, in wait_for return fut.result() File "/opt/lsst/software/stack/miniconda/lib/python3.10/asyncio/streams.py", line 344, in wait_closed await self._protocol._get_close_waiter(self) File "/opt/lsst/software/stack/miniconda/lib/python3.10/site-packages/lsst/ts/salobj/topics/controller_command.py", line 233, in _run_callback ack = await result # type: ignore File "/opt/lsst/software/stack/miniconda/lib/python3.10/site-packages/lsst/ts/m2/csc.py", line 572, in do_disable await self._transition_controller_state( File "/opt/lsst/software/stack/miniconda/lib/python3.10/site-packages/lsst/ts/m2/csc.py", line 630, in _transition_controller_state await self.model.close() File "/opt/lsst/software/stack/miniconda/lib/python3.10/site-packages/lsst/ts/m2/model.py", line 322, in close await self.client_telemetry.close() File "/opt/lsst/software/stack/miniconda/lib/python3.10/site-packages/lsst/ts/m2/tcp_client.py", line 398, in close await self._basic_close() File "/opt/lsst/software/stack/miniconda/lib/python3.10/site-packages/lsst/ts/m2/tcp_client.py", line 251, in _basic_close await tcpip.close_stream_writer(self.writer) File "/opt/lsst/software/stack/miniconda/lib/python3.10/site-packages/lsst/ts/tcpip/utils.py", line 54, in close_stream_writer await asyncio.wait_for(writer.wait_closed(), timeout=5) File "/opt/lsst/software/stack/miniconda/lib/python3.10/asyncio/tasks.py", line 445, in wait_for return fut.result() File "/opt/lsst/software/stack/miniconda/lib/python3.10/asyncio/streams.py", line 344, in wait_closed await self._protocol._get_close_waiter(self) File "/opt/lsst/software/stack/miniconda/lib/python3.10/site-packages/lsst/ts/salobj/topics/controller_command.py", line 233, in _run_callback ack = await result # type: ignore File "/opt/lsst/software/stack/miniconda/lib/python3.10/site-packages/lsst/ts/m2/csc.py", line 696, in do_resetForceOffsets await self._write_command_to_server( File "/opt/lsst/software/stack/miniconda/lib/python3.10/site-packages/lsst/ts/m2/csc.py", line 835, in _write_command_to_server await self.model.close() File "/opt/lsst/software/stack/miniconda/lib/python3.10/site-packages/lsst/ts/m2/model.py", line 322, in close await self.client_telemetry.close() File "/opt/lsst/software/stack/miniconda/lib/python3.10/site-packages/lsst/ts/m2/tcp_client.py", line 398, in close await self._basic_close() File "/opt/lsst/software/stack/miniconda/lib/python3.10/site-packages/lsst/ts/m2/tcp_client.py", line 251, in _basic_close await tcpip.close_stream_writer(self.writer) File "/opt/lsst/software/stack/miniconda/lib/python3.10/site-packages/lsst/ts/tcpip/utils.py", line 54, in close_stream_writer await asyncio.wait_for(writer.wait_closed(), timeout=5) File "/opt/lsst/software/stack/miniconda/lib/python3.10/asyncio/tasks.py", line 445, in wait_for return fut.result() File "/opt/lsst/software/stack/miniconda/lib/python3.10/asyncio/streams.py", line 344, in wait_closed await self._protocol._get_close_waiter(self) File "/opt/lsst/software/stack/miniconda/lib/python3.10/site-packages/lsst/ts/salobj/topics/controller_command.py", line 233, in _run_callback ack = await result # type: ignore File "/opt/lsst/software/stack/miniconda/lib/python3.10/site-packages/lsst/ts/m2/csc.py", line 572, in do_disable await self._transition_controller_state( File "/opt/lsst/software/stack/miniconda/lib/python3.10/site-packages/lsst/ts/m2/csc.py", line 630, in _transition_controller_state await self.model.close() File "/opt/lsst/software/stack/miniconda/lib/python3.10/site-packages/lsst/ts/m2/model.py", line 322, in close await self.client_telemetry.close() File "/opt/lsst/software/stack/miniconda/lib/python3.10/site-packages/lsst/ts/m2/tcp_client.py", line 398, in close await self._basic_close() File "/opt/lsst/software/stack/miniconda/lib/python3.10/site-packages/lsst/ts/m2/tcp_client.py", line 251, in _basic_close await tcpip.close_stream_writer(self.writer) File "/opt/lsst/software/stack/miniconda/lib/python3.10/site-packages/lsst/ts/tcpip/utils.py", line 54, in close_stream_writer await asyncio.wait_for(writer.wait_closed(), timeout=5) File "/opt/lsst/software/stack/miniconda/lib/python3.10/asyncio/tasks.py", line 445, in wait_for return fut.result() File "/opt/lsst/software/stack/miniconda/lib/python3.10/asyncio/streams.py", line 344, in wait_closed await self._protocol._get_close_waiter(self) File "/opt/lsst/software/stack/miniconda/lib/python3.10/site-packages/lsst/ts/m2/csc.py", line 429, in _connection_monitor_loop await self.model.close() File "/opt/lsst/software/stack/miniconda/lib/python3.10/site-packages/lsst/ts/m2/model.py", line 322, in close await self.client_telemetry.close() File "/opt/lsst/software/stack/miniconda/lib/python3.10/site-packages/lsst/ts/m2/tcp_client.py", line 398, in close await self._basic_close() File "/opt/lsst/software/stack/miniconda/lib/python3.10/site-packages/lsst/ts/m2/tcp_client.py", line 251, in _basic_close await tcpip.close_stream_writer(self.writer) File "/opt/lsst/software/stack/miniconda/lib/python3.10/site-packages/lsst/ts/tcpip/utils.py", line 54, in close_stream_writer await asyncio.wait_for(writer.wait_closed(), timeout=5) File "/opt/lsst/software/stack/miniconda/lib/python3.10/asyncio/tasks.py", line 445, in wait_for return fut.result() File "/opt/lsst/software/stack/miniconda/lib/python3.10/asyncio/streams.py", line 344, in wait_closed await self._protocol._get_close_waiter(self) File "/opt/lsst/software/stack/miniconda/lib/python3.10/asyncio/selector_events.py", line 854, in _read_ready_data_received data = self._sock.recv(self.max_size) TimeoutError: [Errno 110] Connection timed out

            bquint Bruno Quint added a comment - - edited Traceback (most recent call last): File "/opt/lsst/software/stack/miniconda/lib/python3.10/site-packages/lsst/ts/m2/csc.py", line 623, in transition_controller_state await self.model.write_command_to_server( File "/opt/lsst/software/stack/miniconda/lib/python3.10/site-packages/lsst/ts/m2/model.py", line 392, in write_command_to_server raise OSError("No TCP/IP connection.") OSError: No TCP/IP connection. During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/opt/lsst/software/stack/miniconda/lib/python3.10/site-packages/lsst/ts/salobj/topics/controller_command.py", line 233, in _run_callback ack = await result # type: ignore File "/opt/lsst/software/stack/miniconda/lib/python3.10/site-packages/lsst/ts/m2/csc.py", line 572, in do_disable await self._transition_controller_state( File "/opt/lsst/software/stack/miniconda/lib/python3.10/site-packages/lsst/ts/m2/csc.py", line 630, in _transition_controller_state await self.model.close() File "/opt/lsst/software/stack/miniconda/lib/python3.10/site-packages/lsst/ts/m2/model.py", line 322, in close await self.client_telemetry.close() File "/opt/lsst/software/stack/miniconda/lib/python3.10/site-packages/lsst/ts/m2/tcp_client.py", line 398, in close await self._basic_close() File "/opt/lsst/software/stack/miniconda/lib/python3.10/site-packages/lsst/ts/m2/tcp_client.py", line 251, in _basic_close await tcpip.close_stream_writer(self.writer) File "/opt/lsst/software/stack/miniconda/lib/python3.10/site-packages/lsst/ts/tcpip/utils.py", line 54, in close_stream_writer await asyncio.wait_for(writer.wait_closed(), timeout=5) File "/opt/lsst/software/stack/miniconda/lib/python3.10/asyncio/tasks.py", line 445, in wait_for return fut.result() File "/opt/lsst/software/stack/miniconda/lib/python3.10/asyncio/streams.py", line 344, in wait_closed await self._protocol._get_close_waiter(self) File "/opt/lsst/software/stack/miniconda/lib/python3.10/site-packages/lsst/ts/salobj/topics/controller_command.py", line 233, in _run_callback ack = await result # type: ignore File "/opt/lsst/software/stack/miniconda/lib/python3.10/site-packages/lsst/ts/m2/csc.py", line 572, in do_disable await self._transition_controller_state( File "/opt/lsst/software/stack/miniconda/lib/python3.10/site-packages/lsst/ts/m2/csc.py", line 630, in _transition_controller_state await self.model.close() File "/opt/lsst/software/stack/miniconda/lib/python3.10/site-packages/lsst/ts/m2/model.py", line 322, in close await self.client_telemetry.close() File "/opt/lsst/software/stack/miniconda/lib/python3.10/site-packages/lsst/ts/m2/tcp_client.py", line 398, in close await self._basic_close() File "/opt/lsst/software/stack/miniconda/lib/python3.10/site-packages/lsst/ts/m2/tcp_client.py", line 251, in _basic_close await tcpip.close_stream_writer(self.writer) File "/opt/lsst/software/stack/miniconda/lib/python3.10/site-packages/lsst/ts/tcpip/utils.py", line 54, in close_stream_writer await asyncio.wait_for(writer.wait_closed(), timeout=5) File "/opt/lsst/software/stack/miniconda/lib/python3.10/asyncio/tasks.py", line 445, in wait_for return fut.result() File "/opt/lsst/software/stack/miniconda/lib/python3.10/asyncio/streams.py", line 344, in wait_closed await self._protocol._get_close_waiter(self) File "/opt/lsst/software/stack/miniconda/lib/python3.10/site-packages/lsst/ts/salobj/topics/controller_command.py", line 233, in _run_callback ack = await result # type: ignore File "/opt/lsst/software/stack/miniconda/lib/python3.10/site-packages/lsst/ts/m2/csc.py", line 572, in do_disable await self._transition_controller_state( File "/opt/lsst/software/stack/miniconda/lib/python3.10/site-packages/lsst/ts/m2/csc.py", line 630, in _transition_controller_state await self.model.close() File "/opt/lsst/software/stack/miniconda/lib/python3.10/site-packages/lsst/ts/m2/model.py", line 322, in close await self.client_telemetry.close() File "/opt/lsst/software/stack/miniconda/lib/python3.10/site-packages/lsst/ts/m2/tcp_client.py", line 398, in close await self._basic_close() File "/opt/lsst/software/stack/miniconda/lib/python3.10/site-packages/lsst/ts/m2/tcp_client.py", line 251, in _basic_close await tcpip.close_stream_writer(self.writer) File "/opt/lsst/software/stack/miniconda/lib/python3.10/site-packages/lsst/ts/tcpip/utils.py", line 54, in close_stream_writer await asyncio.wait_for(writer.wait_closed(), timeout=5) File "/opt/lsst/software/stack/miniconda/lib/python3.10/asyncio/tasks.py", line 445, in wait_for return fut.result() File "/opt/lsst/software/stack/miniconda/lib/python3.10/asyncio/streams.py", line 344, in wait_closed await self._protocol._get_close_waiter(self) File "/opt/lsst/software/stack/miniconda/lib/python3.10/site-packages/lsst/ts/salobj/topics/controller_command.py", line 233, in _run_callback ack = await result # type: ignore File "/opt/lsst/software/stack/miniconda/lib/python3.10/site-packages/lsst/ts/m2/csc.py", line 572, in do_disable await self._transition_controller_state( File "/opt/lsst/software/stack/miniconda/lib/python3.10/site-packages/lsst/ts/m2/csc.py", line 630, in _transition_controller_state await self.model.close() File "/opt/lsst/software/stack/miniconda/lib/python3.10/site-packages/lsst/ts/m2/model.py", line 322, in close await self.client_telemetry.close() File "/opt/lsst/software/stack/miniconda/lib/python3.10/site-packages/lsst/ts/m2/tcp_client.py", line 398, in close await self._basic_close() File "/opt/lsst/software/stack/miniconda/lib/python3.10/site-packages/lsst/ts/m2/tcp_client.py", line 251, in _basic_close await tcpip.close_stream_writer(self.writer) File "/opt/lsst/software/stack/miniconda/lib/python3.10/site-packages/lsst/ts/tcpip/utils.py", line 54, in close_stream_writer await asyncio.wait_for(writer.wait_closed(), timeout=5) File "/opt/lsst/software/stack/miniconda/lib/python3.10/asyncio/tasks.py", line 445, in wait_for return fut.result() File "/opt/lsst/software/stack/miniconda/lib/python3.10/asyncio/streams.py", line 344, in wait_closed await self._protocol._get_close_waiter(self) File "/opt/lsst/software/stack/miniconda/lib/python3.10/site-packages/lsst/ts/salobj/topics/controller_command.py", line 233, in _run_callback ack = await result # type: ignore File "/opt/lsst/software/stack/miniconda/lib/python3.10/site-packages/lsst/ts/m2/csc.py", line 696, in do_resetForceOffsets await self._write_command_to_server( File "/opt/lsst/software/stack/miniconda/lib/python3.10/site-packages/lsst/ts/m2/csc.py", line 835, in _write_command_to_server await self.model.close() File "/opt/lsst/software/stack/miniconda/lib/python3.10/site-packages/lsst/ts/m2/model.py", line 322, in close await self.client_telemetry.close() File "/opt/lsst/software/stack/miniconda/lib/python3.10/site-packages/lsst/ts/m2/tcp_client.py", line 398, in close await self._basic_close() File "/opt/lsst/software/stack/miniconda/lib/python3.10/site-packages/lsst/ts/m2/tcp_client.py", line 251, in _basic_close await tcpip.close_stream_writer(self.writer) File "/opt/lsst/software/stack/miniconda/lib/python3.10/site-packages/lsst/ts/tcpip/utils.py", line 54, in close_stream_writer await asyncio.wait_for(writer.wait_closed(), timeout=5) File "/opt/lsst/software/stack/miniconda/lib/python3.10/asyncio/tasks.py", line 445, in wait_for return fut.result() File "/opt/lsst/software/stack/miniconda/lib/python3.10/asyncio/streams.py", line 344, in wait_closed await self._protocol._get_close_waiter(self) File "/opt/lsst/software/stack/miniconda/lib/python3.10/site-packages/lsst/ts/salobj/topics/controller_command.py", line 233, in _run_callback ack = await result # type: ignore File "/opt/lsst/software/stack/miniconda/lib/python3.10/site-packages/lsst/ts/m2/csc.py", line 572, in do_disable await self._transition_controller_state( File "/opt/lsst/software/stack/miniconda/lib/python3.10/site-packages/lsst/ts/m2/csc.py", line 630, in _transition_controller_state await self.model.close() File "/opt/lsst/software/stack/miniconda/lib/python3.10/site-packages/lsst/ts/m2/model.py", line 322, in close await self.client_telemetry.close() File "/opt/lsst/software/stack/miniconda/lib/python3.10/site-packages/lsst/ts/m2/tcp_client.py", line 398, in close await self._basic_close() File "/opt/lsst/software/stack/miniconda/lib/python3.10/site-packages/lsst/ts/m2/tcp_client.py", line 251, in _basic_close await tcpip.close_stream_writer(self.writer) File "/opt/lsst/software/stack/miniconda/lib/python3.10/site-packages/lsst/ts/tcpip/utils.py", line 54, in close_stream_writer await asyncio.wait_for(writer.wait_closed(), timeout=5) File "/opt/lsst/software/stack/miniconda/lib/python3.10/asyncio/tasks.py", line 445, in wait_for return fut.result() File "/opt/lsst/software/stack/miniconda/lib/python3.10/asyncio/streams.py", line 344, in wait_closed await self._protocol._get_close_waiter(self) File "/opt/lsst/software/stack/miniconda/lib/python3.10/site-packages/lsst/ts/m2/csc.py", line 429, in _connection_monitor_loop await self.model.close() File "/opt/lsst/software/stack/miniconda/lib/python3.10/site-packages/lsst/ts/m2/model.py", line 322, in close await self.client_telemetry.close() File "/opt/lsst/software/stack/miniconda/lib/python3.10/site-packages/lsst/ts/m2/tcp_client.py", line 398, in close await self._basic_close() File "/opt/lsst/software/stack/miniconda/lib/python3.10/site-packages/lsst/ts/m2/tcp_client.py", line 251, in _basic_close await tcpip.close_stream_writer(self.writer) File "/opt/lsst/software/stack/miniconda/lib/python3.10/site-packages/lsst/ts/tcpip/utils.py", line 54, in close_stream_writer await asyncio.wait_for(writer.wait_closed(), timeout=5) File "/opt/lsst/software/stack/miniconda/lib/python3.10/asyncio/tasks.py", line 445, in wait_for return fut.result() File "/opt/lsst/software/stack/miniconda/lib/python3.10/asyncio/streams.py", line 344, in wait_closed await self._protocol._get_close_waiter(self) File "/opt/lsst/software/stack/miniconda/lib/python3.10/asyncio/selector_events.py", line 854, in _read_ready _data_received data = self._sock.recv(self.max_size) TimeoutError: [Errno 110] Connection timed out
            ttsai Te-Wei Tsai added a comment - - edited

            bquint Please try the ts_m2 instead if you think it is from M2 CSC. I do not know the mtcs at all. Thanks! 

            Based on the error message, you can see the connection timeout. Therefore, you need to check  what is the host name and ports of M2 at this moment. I think it should be reported in the log.

            ttsai Te-Wei Tsai added a comment - - edited bquint  Please try the ts_m2  instead if you think it is from M2 CSC. I do not know the mtcs at all. Thanks!  Based on the error message, you can see the connection timeout. Therefore, you need to check  what is the host name and ports of M2 at this moment. I think it should be reported in the log.
            tribeiro Tiago Ribeiro added a comment - - edited

            ttsai, please, look at the traceback Bruno sent on the message above... you can ignore the MTCS error.

            The traceback above comes from the CSC itself. What I think we need to do is add "try/except" clauses around potential failures and ignore them when doing the transition from ENABLED to DISABLED, and DISABLED to STANDBY, while doing what is needed to cleanup any pending connection.

            In the case above, I think we want to protect this call from failure, as you can see from:

             await self._transition_controller_state( File "/opt/lsst/software/stack/miniconda/lib/python3.10/site-packages/lsst/ts/m2/csc.py", line 630
            

            tribeiro Tiago Ribeiro added a comment - - edited ttsai , please, look at the traceback Bruno sent on the message above... you can ignore the MTCS error. The traceback above comes from the CSC itself. What I think we need to do is add "try/except" clauses around potential failures and ignore them when doing the transition from ENABLED to DISABLED, and DISABLED to STANDBY, while doing what is needed to cleanup any pending connection. In the case above, I think we want to protect this call from failure, as you can see from: await self._transition_controller_state( File "/opt/lsst/software/stack/miniconda/lib/python3.10/site-packages/lsst/ts/m2/csc.py", line 630
            ttsai Te-Wei Tsai added a comment - - edited

            This is interesting. I am not sure this is right or not. The error message is connection timeout. Therefore, I am guessing the hostname or ports are not right. They are configured here:

            https://github.com/lsst-ts/ts_config_mttcs/tree/develop/MTM2/v2

            There is the log message here to check:
            https://github.com/lsst-ts/ts_m2/blob/ef85c2f2413a4479db97931ae10be389771795e6/python/lsst/ts/m2/csc.py#L486-L488

            Then, if the server closed the connection, the CSC will transition to the Fault state actually:
            https://github.com/lsst-ts/ts_m2/blob/ef85c2f2413a4479db97931ae10be389771795e6/python/lsst/ts/m2/csc.py#L408-L416

            In addition, if for some how the above function does not work, there should be no telemetry at that time. And there is the related warning message here:

            https://github.com/lsst-ts/ts_m2/blob/ef85c2f2413a4479db97931ae10be389771795e6/python/lsst/ts/m2/csc.py#L365-L371

            Therefore, it is weird to me actually.

            ttsai Te-Wei Tsai added a comment - - edited This is interesting. I am not sure this is right or not. The error message is connection timeout. Therefore, I am guessing the hostname or ports are not right. They are configured here: https://github.com/lsst-ts/ts_config_mttcs/tree/develop/MTM2/v2 There is the log message here to check: https://github.com/lsst-ts/ts_m2/blob/ef85c2f2413a4479db97931ae10be389771795e6/python/lsst/ts/m2/csc.py#L486-L488 Then, if the server closed the connection, the CSC will transition to the Fault state actually: https://github.com/lsst-ts/ts_m2/blob/ef85c2f2413a4479db97931ae10be389771795e6/python/lsst/ts/m2/csc.py#L408-L416 In addition, if for some how the above function does not work, there should be no telemetry at that time. And there is the related warning message here: https://github.com/lsst-ts/ts_m2/blob/ef85c2f2413a4479db97931ae10be389771795e6/python/lsst/ts/m2/csc.py#L365-L371 Therefore, it is weird to me actually.
            ttsai Te-Wei Tsai added a comment -

            I think it is better to check the log message first to see any clue to know what happened at that time.

            ttsai Te-Wei Tsai added a comment - I think it is better to check the log message first to see any clue to know what happened at that time.
            ttsai Te-Wei Tsai added a comment -

            I will debug this at DM-35454 to use the story point together.

            ttsai Te-Wei Tsai added a comment - I will debug this at DM-35454 to use the story point together.
            ttsai Te-Wei Tsai added a comment -

            bquint Please help to provide the related timestamp for me to query the log messages at the EFD. Thanks! This is at the Tucson Teststand instead of summit. Right?

            ttsai Te-Wei Tsai added a comment - bquint  Please help to provide the related timestamp for me to query the log messages at the EFD. Thanks! This is at the Tucson Teststand instead of summit. Right?
            ttsai Te-Wei Tsai added a comment - Some similar discussion here: https://github.com/home-assistant/core/issues/10468
            ttsai Te-Wei Tsai added a comment -

            Please help to review the PR:
            https://github.com/lsst-ts/ts_m2/pull/27

            Thanks!

            ttsai Te-Wei Tsai added a comment - Please help to review the PR: https://github.com/lsst-ts/ts_m2/pull/27 Thanks!
            ttsai Te-Wei Tsai added a comment -

            This should be resolved in DM-35454.

            ttsai Te-Wei Tsai added a comment - This should be resolved in DM-35454 .

            People

              ttsai Te-Wei Tsai
              bquint Bruno Quint
              Tiago Ribeiro
              Bruno Quint, Te-Wei Tsai, Tiago Ribeiro
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Jenkins

                  No builds found.