[DM-28687] The MTHexapod low-level controller mis-handles clearError if there is a fault condition - Jira

XML

Word

Printable

Details

Type: Story
Status: Done
Resolution: Done
Fix Version/s: None
Component/s: ts_main_telescope
Labels:
- HexRot
- MainTelescope

Story Points:
4
Epic Link:
Hexapod Required Changes
Sprint:
TSSW Sprint - Jun 21 - Jul 05, TSSW Sprint - Jul 05 - Jul 19
Team:
Telescope and Site
Urgent?:
No

Description

If there is a fault condition such as an e-stop pressed then the clearError command oscillates between the "FAULT" and "STANDBY" state forever. I think it should got to FAULT and stay in FAULT.

I observed this in today's testing. I believe the sequence was:

Use the EUI to put the controller into DDS mode
Use the hexapod commander to issue "enterControl" to the CSC
The state goes to FAULT
Use the hexapod commander to issue "clearError" to the CSC
The state oscillates between FAULT and STANDBY. I see no sign that it will ever stop doing this.

That's the core of the problem above. But we continued as follows:

Use the hexapod commander to issue "exitControl" to the CSC. The oscillation stopped.
Use the EUI to issue "clearError". The oscillation resumed (as reported to the commander).

Attachments

Options
- Sort By Name
- Sort By Date
- Ascending
- Descending
- Thumbnails
- List
- Download All

Attachments

dataInEfd.png
513 kB
13/Jul/21 9:29 PM
enabledStateWithInvalidCommand.png
417 kB
13/Jul/21 9:29 PM
IMG_20210712_140632.jpg
618 kB
12/Jul/21 6:16 PM
stateMachine.png
91 kB
13/Jul/21 8:18 PM
stateTransition.png
431 kB
09/Jul/21 7:03 PM

Issue Links

relates to

DM-29931 The camera hexapod starts to move when DDS control is switched on.

To Do

DM-28695 The MTHexapod is mis-handling configuration changes

Invalid

DM-29975 Please make the STOP command more reliable in the hexapod

To Do

DM-30814 Have the Rotator Send Exactly the Same Telemetry and Config Data to the EUI and CSC

Done

DM-29720 Support the Command Queue in Low-Level Hexapod Controller

Invalid

Activity

Ascending order - Click to sort in descending order

No builds found.

Russell Owen created issue - 05/Feb/21 6:23 PM

Te-Wei Tsai made changes - 05/Feb/21 8:44 PM

Field	Original Value	New Value
Epic Link		~~DM-27621~~ [ 441885 ]

Te-Wei Tsai made changes - 05/Feb/21 8:45 PM

Labels

HexRot MainTelescope

Russell Owen made changes - 05/Feb/21 11:13 PM

Link

This issue relates to ~~DM-28695~~ [ ~~DM-28695~~ ]

Russell Owen made changes - 05/Feb/21 11:14 PM

Comment

[ Also (perhaps this should be a different ticket) changing the configuration does not trigger a new configuration message from the low-level controller. [~pkubanek] changed the pivot point in the low-level controller and the CSC apparently did not see a configuration message (certainly it did not print new configuration, which it does when it sees a configuration message).

Also a question: are the commands to set the pivot, acceleration, etc. only accepted in OFFLINE state? I was not able to change them from the CSC in ENABLED state (and the CSC rejects those commands in any other state, of course). ]

Hide

Permalink

Te-Wei Tsai added a comment - 05/Feb/21 11:55 PM - edited

Based on the log file, the following messages were recoded:

Feb  5 11:06:12 localhost journal: LSST Wrapper: new mode= 4.000000

Feb  5 11:06:12 localhost journal: LSST Wrapper: new mode= 0.000000

Feb  5 11:06:12 localhost journal: LSST Wrapper: new mode= 4.000000

Feb  5 11:06:12 localhost journal: LSST Wrapper: new mode= 0.000000

Feb  5 11:06:12 localhost journal: LSST Wrapper: STO interlock open

Feb  5 11:05:47 localhost journal: LSST Wrapper: new mode= 4.000000

Feb  5 11:05:47 localhost journal: LSST Wrapper: new mode= 0.000000

Feb  5 11:05:32 localhost journal: LSST Wrapper: Command source now == DDS

There is the continuous oscillation of modes of 0 and 4. Interesting thing is "STO interlock open".

There is no difference of clearing error code in hexapod controller between the EUI and DDS.

Show

Te-Wei Tsai added a comment - 05/Feb/21 11:55 PM - edited Based on the log file, the following messages were recoded: Feb 5 11:06:12 localhost journal: LSST Wrapper: new mode= 4.000000 Feb 5 11:06:12 localhost journal: LSST Wrapper: new mode= 0.000000 Feb 5 11:06:12 localhost journal: LSST Wrapper: new mode= 4.000000 Feb 5 11:06:12 localhost journal: LSST Wrapper: new mode= 0.000000 Feb 5 11:06:12 localhost journal: LSST Wrapper: STO interlock open Feb 5 11:05:47 localhost journal: LSST Wrapper: new mode= 4.000000 Feb 5 11:05:47 localhost journal: LSST Wrapper: new mode= 0.000000 Feb 5 11:05:32 localhost journal: LSST Wrapper: Command source now == DDS There is the continuous oscillation of modes of 0 and 4. Interesting thing is "STO interlock open". There is no difference of clearing error code in hexapod controller between the EUI and DDS.

Tim Jenness made changes - 16/Feb/21 10:27 PM

Team

Telescope and Site [ 13500 ]

Te-Wei Tsai made changes - 17/Jun/21 9:04 PM

End date		25/Jun/21
Sprint		TSSW Sprint - Jun 21 - Jul 05 [ 1104 ]
Start date		24/Jun/21
Story Points		1

Hide

Permalink

Russell Owen added a comment - 17/Jun/21 9:28 PM

Is "STO interlock open" simply the indication that an e-stop was pressed? If so I agree it is interesting that it was reported during this flood of mode=4/mode=0 since that did not change.

Show

Russell Owen added a comment - 17/Jun/21 9:28 PM Is "STO interlock open" simply the indication that an e-stop was pressed? If so I agree it is interesting that it was reported during this flood of mode=4/mode=0 since that did not change.

Te-Wei Tsai made changes - 24/Jun/21 6:45 PM

Status

To Do [ 10001 ]

In Progress [ 3 ]

Te-Wei Tsai made changes - 24/Jun/21 6:58 PM

Status

In Progress [ 3 ]

To Do [ 10001 ]

Te-Wei Tsai made changes - 06/Jul/21 6:04 PM

Status

To Do [ 10001 ]

In Progress [ 3 ]

Hide

Permalink

Te-Wei Tsai added a comment - 06/Jul/21 8:48 PM - edited

Based on the following code (in drive.c), it looks like the interlock was on (or open) in the test:

    if (i < 2) {

        if ((din->copleyInputPins & INTERLOCK_MASK) != lastInterlockPins) {

            // is there an open interlock circuit?

            if ((din->copleyInputPins & INTERLOCK_MASK) != INTERLOCK_MASK) {

                if (!gInterlockOpen) {

                    syslog(LOG_ERR, "%s", "STO interlock open");

                    gInterlockOpen = 1;

            } else {

                if (gInterlockOpen)

                    syslog(LOG_NOTICE, "STO circuit OK");

                gInterlockOpen = 0;

Assume the interlock is opened, the system would be in the Fault state. When we try to clear the error, the system should go to the Standby state first. Then, the drive.c in the low-level controller would find the interlock opens, and put the system back to the Fault state. This means we should only see Fault -> Standby -> Fault instead of continuous oscillation.

Therefore, it looks like for somehow, the ClearError in low-level controller is triggered continuously. At each time after the system transitions to the Standby state, the existed interlock opening transitions the system back to the Fault state.

Show

Te-Wei Tsai added a comment - 06/Jul/21 8:48 PM - edited Based on the following code (in drive.c ), it looks like the interlock was on (or open) in the test: if (i < 2) { if ((din->copleyInputPins & INTERLOCK_MASK) != lastInterlockPins) { // is there an open interlock circuit? if ((din->copleyInputPins & INTERLOCK_MASK) != INTERLOCK_MASK) { if (!gInterlockOpen) { syslog(LOG_ERR, "%s" , "STO interlock open" ); gInterlockOpen = 1; } } else { if (gInterlockOpen) syslog(LOG_NOTICE, "STO circuit OK" ); gInterlockOpen = 0; } } } Assume the interlock is opened, the system would be in the Fault state. When we try to clear the error, the system should go to the Standby state first. Then, the drive.c in the low-level controller would find the interlock opens, and put the system back to the Fault state. This means we should only see Fault -> Standby -> Fault instead of continuous oscillation. Therefore, it looks like for somehow, the ClearError in low-level controller is triggered continuously. At each time after the system transitions to the Standby state, the existed interlock opening transitions the system back to the Fault state.

Hide

Permalink

Te-Wei Tsai added a comment - 06/Jul/21 9:53 PM - edited

I think the problem might come from the read and write of cmdMsgBuffer and gCmdMsgBuffInd among thread (ddsCmdSocket.c, cmdClientSocket.c, and commanding.c).

Show

Te-Wei Tsai added a comment - 06/Jul/21 9:53 PM - edited I think the problem might come from the read and write of cmdMsgBuffer and gCmdMsgBuffInd among thread ( ddsCmdSocket.c , cmdClientSocket.c , and commanding.c ).

Wouter van Reeven made changes - 07/Jul/21 2:56 PM

Sprint

TSSW Sprint - Jun 21 - Jul 05 [ 1104 ]

TSSW Sprint - Jun 21 - Jul 05, TSSW Sprint - Jul 5 - Jul 19 [ 1104, 1106 ]

Hide

Permalink

Te-Wei Tsai added a comment - 07/Jul/21 3:41 PM - edited

The DDS and GUI can not command the Camera Hexapod in the same time. Therefore, there should be no race condition between ddsCmdSocket.c and cmdClientSocket.c.

In the commanding.c, the code will increase the gCmdMsgBuffInd by itself in some conditions. However, it does not check the limit of buffer (cmdMsgBuffer). Therefore, I think this might be the reason to have some unpredicted behavior happened.

Show

Te-Wei Tsai added a comment - 07/Jul/21 3:41 PM - edited The DDS and GUI can not command the Camera Hexapod in the same time. Therefore, there should be no race condition between ddsCmdSocket.c and cmdClientSocket.c . In the commanding.c , the code will increase the gCmdMsgBuffInd by itself in some conditions. However, it does not check the limit of buffer ( cmdMsgBuffer ). Therefore, I think this might be the reason to have some unpredicted behavior happened.

Te-Wei Tsai made changes - 08/Jul/21 5:55 PM

Story Points

Hide

Permalink

Te-Wei Tsai added a comment - 08/Jul/21 7:04 PM - edited

Added a thread-safe circular buffer for the command. Replaced the command message buffer to use the thread-safe queue.

Show

Te-Wei Tsai added a comment - 08/Jul/21 7:04 PM - edited Added a thread-safe circular buffer for the command. Replaced the command message buffer to use the thread-safe queue.

Te-Wei Tsai made changes - 09/Jul/21 7:03 PM

Attachment

stateTransition.png [ 50955 ]

Hide

Permalink

Te-Wei Tsai added a comment - 09/Jul/21 7:04 PM

Upgraded the software on summit and the state transition by GUI can work correctly:

For the log message:

Jul  9 18:48:07 localhost journal: LSST Wrapper: new mode= 2.000000

Jul  9 18:48:07 localhost journal: LSST Wrapper: processed State Cmd: 0x5555, 0x8000, counter=5, parameter1 = 2.000000.

Jul  9 18:47:52 localhost journal: LSST Wrapper: new mode= 1.000000

Jul  9 18:47:52 localhost journal: LSST Wrapper: processed State Cmd: 0x5555, 0x8000, counter=4, parameter1 = 1.000000.

Jul  9 18:47:46 localhost journal: LSST Wrapper: processed State Cmd: 0x5555, 0x8000, counter=3, parameter1 = 4.000000.

Jul  9 18:47:35 localhost journal: LSST Wrapper: new mode= 0.000000

Jul  9 18:47:35 localhost journal: LSST Wrapper: processed State Cmd: 0x5555, 0x8000, counter=2, parameter1 = 7.000000.

Show

Te-Wei Tsai added a comment - 09/Jul/21 7:04 PM Upgraded the software on summit and the state transition by GUI can work correctly: For the log message: Jul 9 18:48:07 localhost journal: LSST Wrapper: new mode= 2.000000 Jul 9 18:48:07 localhost journal: LSST Wrapper: processed State Cmd: 0x5555, 0x8000, counter=5, parameter1 = 2.000000. Jul 9 18:47:52 localhost journal: LSST Wrapper: new mode= 1.000000 Jul 9 18:47:52 localhost journal: LSST Wrapper: processed State Cmd: 0x5555, 0x8000, counter=4, parameter1 = 1.000000. Jul 9 18:47:46 localhost journal: LSST Wrapper: processed State Cmd: 0x5555, 0x8000, counter=3, parameter1 = 4.000000. Jul 9 18:47:35 localhost journal: LSST Wrapper: new mode= 0.000000 Jul 9 18:47:35 localhost journal: LSST Wrapper: processed State Cmd: 0x5555, 0x8000, counter=2, parameter1 = 7.000000.

Te-Wei Tsai made changes - 09/Jul/21 7:08 PM

Link

This issue relates to ~~DM-29720~~ [ ~~DM-29720~~ ]

Te-Wei Tsai made changes - 09/Jul/21 7:23 PM

Link

This issue relates to DM-29975 [ DM-29975 ]

Te-Wei Tsai made changes - 09/Jul/21 7:27 PM

Link

This issue relates to DM-29931 [ DM-29931 ]

Hide

Permalink

Te-Wei Tsai added a comment - 09/Jul/21 9:38 PM

Please help to review the PR:
https://github.com/lsst-ts/ts_hexapod_controller/pull/19

Thanks!

Show

Te-Wei Tsai added a comment - 09/Jul/21 9:38 PM Please help to review the PR: https://github.com/lsst-ts/ts_hexapod_controller/pull/19 Thanks!

Te-Wei Tsai made changes - 09/Jul/21 9:38 PM

Reviewers		Russell Owen [ rowen ]
Status	In Progress [ 3 ]	In Review [ 10004 ]

Hide

Permalink

Te-Wei Tsai added a comment - 12/Jul/21 4:39 PM - edited

Holger Drass did the test on summit today and he could not clear the interlock fault. Before we checked with the hardware engineer the interlock button is pressed or not, we tried to clear the error by the software. And we could see the oscillation of system between the standby state and fault state. This is very strange because the the thread-safe circular buffer is used already. Therefore, there should be no overflow or race condition now. I began to suspect this comes from the Simulink model:

ul 12 16:23:52 localhost journal: LSST Wrapper: new mode= 0.000000

Jul 12 16:23:52 localhost journal: LSST Wrapper: new mode= 4.000000

Jul 12 16:23:52 localhost journal: LSST Wrapper: new mode= 0.000000

Jul 12 16:23:52 localhost journal: LSST Wrapper: new mode= 4.000000

Jul 12 16:23:52 localhost journal: LSST Wrapper: new mode= 0.000000

Jul 12 16:23:52 localhost journal: LSST Wrapper: new mode= 4.000000

Jul 12 16:23:52 localhost journal: LSST Wrapper: new mode= 0.000000

Jul 12 16:23:52 localhost journal: LSST Wrapper: new mode= 4.000000

Jul 12 16:23:52 localhost journal: LSST Wrapper: Clearing drive faults.

Jul 12 16:23:52 localhost journal: LSST Wrapper: processed State Cmd: 0x5555, 0x8000, counter=4, parameter1 = 6.000000.

Jul 12 16:23:52 localhost journal: LSST Wrapper: STO interlock open

Jul 12 16:23:52 localhost journal: LSST Wrapper: new mode= 0.000000

Jul 12 16:23:52 localhost journal: LSST Wrapper: new mode= 4.000000

Jul 12 16:23:52 localhost journal: LSST Wrapper: new mode= 0.000000

Jul 12 16:23:52 localhost journal: LSST Wrapper: new mode= 4.000000

Jul 12 16:23:52 localhost journal: LSST Wrapper: new mode= 0.000000

Jul 12 16:23:52 localhost journal: LSST Wrapper: new mode= 4.000000

Jul 12 16:23:52 localhost journal: LSST Wrapper: new mode= 0.000000

Jul 12 16:23:52 localhost journal: LSST Wrapper: new mode= 4.000000

Jul 12 16:23:52 localhost journal: LSST Wrapper: new mode= 0.000000

Jul 12 16:23:52 localhost journal: LSST Wrapper: new mode= 4.000000

--

Jul 12 16:23:52 localhost journal: LSST Wrapper: new mode= 0.000000

Jul 12 16:23:52 localhost journal: LSST Wrapper: new mode= 4.000000

Jul 12 16:23:52 localhost journal: LSST Wrapper: new mode= 0.000000

Jul 12 16:23:52 localhost journal: LSST Wrapper: new mode= 4.000000

Jul 12 16:23:52 localhost journal: LSST Wrapper: new mode= 0.000000

Jul 12 16:23:52 localhost journal: LSST Wrapper: new mode= 4.000000

Jul 12 16:23:52 localhost journal: LSST Wrapper: new mode= 0.000000

Jul 12 16:23:52 localhost journal: LSST Wrapper: new mode= 4.000000

Jul 12 16:23:52 localhost journal: LSST Wrapper: Clearing drive faults.

Jul 12 16:23:52 localhost journal: LSST Wrapper: processed State Cmd: 0x5555, 0x8000, counter=4, parameter1 = 6.000000.

Jul 12 16:23:52 localhost journal: LSST Wrapper: STO interlock open

Jul 12 16:23:52 localhost journal: LSST Wrapper: new mode= 0.000000

Jul 12 16:23:52 localhost journal: LSST Wrapper: new mode= 4.000000

Jul 12 16:23:52 localhost journal: LSST Wrapper: new mode= 0.000000

Jul 12 16:23:52 localhost journal: LSST Wrapper: new mode= 4.000000

Jul 12 16:23:52 localhost journal: LSST Wrapper: new mode= 0.000000

Jul 12 16:23:52 localhost journal: LSST Wrapper: new mode= 4.000000

Jul 12 16:23:52 localhost journal: LSST Wrapper: new mode= 0.000000

Jul 12 16:23:52 localhost journal: LSST Wrapper: new mode= 4.000000

Jul 12 16:23:52 localhost journal: LSST Wrapper: new mode= 0.000000

Jul 12 16:23:52 localhost journal: LSST Wrapper: new mode= 4.000000

Update from Holger:
Mario helped me to clear the interlock fault.
The bottom was not pressed.
He needed to clear the interlock and then send a reset through a two button connected to the hexapod electronics cabinet.

Show

Te-Wei Tsai added a comment - 12/Jul/21 4:39 PM - edited Holger Drass did the test on summit today and he could not clear the interlock fault. Before we checked with the hardware engineer the interlock button is pressed or not, we tried to clear the error by the software. And we could see the oscillation of system between the standby state and fault state. This is very strange because the the thread-safe circular buffer is used already. Therefore, there should be no overflow or race condition now. I began to suspect this comes from the Simulink model: ul 12 16:23:52 localhost journal: LSST Wrapper: new mode= 0.000000 Jul 12 16:23:52 localhost journal: LSST Wrapper: new mode= 4.000000 Jul 12 16:23:52 localhost journal: LSST Wrapper: new mode= 0.000000 Jul 12 16:23:52 localhost journal: LSST Wrapper: new mode= 4.000000 Jul 12 16:23:52 localhost journal: LSST Wrapper: new mode= 0.000000 Jul 12 16:23:52 localhost journal: LSST Wrapper: new mode= 4.000000 Jul 12 16:23:52 localhost journal: LSST Wrapper: new mode= 0.000000 Jul 12 16:23:52 localhost journal: LSST Wrapper: new mode= 4.000000 Jul 12 16:23:52 localhost journal: LSST Wrapper: Clearing drive faults. Jul 12 16:23:52 localhost journal: LSST Wrapper: processed State Cmd: 0x5555, 0x8000, counter=4, parameter1 = 6.000000. Jul 12 16:23:52 localhost journal: LSST Wrapper: STO interlock open Jul 12 16:23:52 localhost journal: LSST Wrapper: new mode= 0.000000 Jul 12 16:23:52 localhost journal: LSST Wrapper: new mode= 4.000000 Jul 12 16:23:52 localhost journal: LSST Wrapper: new mode= 0.000000 Jul 12 16:23:52 localhost journal: LSST Wrapper: new mode= 4.000000 Jul 12 16:23:52 localhost journal: LSST Wrapper: new mode= 0.000000 Jul 12 16:23:52 localhost journal: LSST Wrapper: new mode= 4.000000 Jul 12 16:23:52 localhost journal: LSST Wrapper: new mode= 0.000000 Jul 12 16:23:52 localhost journal: LSST Wrapper: new mode= 4.000000 Jul 12 16:23:52 localhost journal: LSST Wrapper: new mode= 0.000000 Jul 12 16:23:52 localhost journal: LSST Wrapper: new mode= 4.000000 -- Jul 12 16:23:52 localhost journal: LSST Wrapper: new mode= 0.000000 Jul 12 16:23:52 localhost journal: LSST Wrapper: new mode= 4.000000 Jul 12 16:23:52 localhost journal: LSST Wrapper: new mode= 0.000000 Jul 12 16:23:52 localhost journal: LSST Wrapper: new mode= 4.000000 Jul 12 16:23:52 localhost journal: LSST Wrapper: new mode= 0.000000 Jul 12 16:23:52 localhost journal: LSST Wrapper: new mode= 4.000000 Jul 12 16:23:52 localhost journal: LSST Wrapper: new mode= 0.000000 Jul 12 16:23:52 localhost journal: LSST Wrapper: new mode= 4.000000 Jul 12 16:23:52 localhost journal: LSST Wrapper: Clearing drive faults. Jul 12 16:23:52 localhost journal: LSST Wrapper: processed State Cmd: 0x5555, 0x8000, counter=4, parameter1 = 6.000000. Jul 12 16:23:52 localhost journal: LSST Wrapper: STO interlock open Jul 12 16:23:52 localhost journal: LSST Wrapper: new mode= 0.000000 Jul 12 16:23:52 localhost journal: LSST Wrapper: new mode= 4.000000 Jul 12 16:23:52 localhost journal: LSST Wrapper: new mode= 0.000000 Jul 12 16:23:52 localhost journal: LSST Wrapper: new mode= 4.000000 Jul 12 16:23:52 localhost journal: LSST Wrapper: new mode= 0.000000 Jul 12 16:23:52 localhost journal: LSST Wrapper: new mode= 4.000000 Jul 12 16:23:52 localhost journal: LSST Wrapper: new mode= 0.000000 Jul 12 16:23:52 localhost journal: LSST Wrapper: new mode= 4.000000 Jul 12 16:23:52 localhost journal: LSST Wrapper: new mode= 0.000000 Jul 12 16:23:52 localhost journal: LSST Wrapper: new mode= 4.000000 Update from Holger: Mario helped me to clear the interlock fault. The bottom was not pressed. He needed to clear the interlock and then send a reset through a two button connected to the hexapod electronics cabinet.

Te-Wei Tsai made changes - 12/Jul/21 4:47 PM

Status

In Review [ 10004 ]

In Progress [ 3 ]

Hide

Permalink

Te-Wei Tsai added a comment - 12/Jul/21 4:48 PM

I closed the original PR first because I suspected there is the problem of Simulink model as well.

Show

Te-Wei Tsai added a comment - 12/Jul/21 4:48 PM I closed the original PR first because I suspected there is the problem of Simulink model as well.

Te-Wei Tsai made changes - 12/Jul/21 4:48 PM

Story Points

Holger Drass made changes - 12/Jul/21 6:16 PM

Attachment

IMG_20210712_140632.jpg [ 50978 ]

Hide

Permalink

Holger Drass added a comment - 12/Jul/21 6:17 PM

Thanks Mario Rivera! Pictures on the buttons added.

Show

Holger Drass added a comment - 12/Jul/21 6:17 PM Thanks Mario Rivera ! Pictures on the buttons added.

Russell Owen made changes - 12/Jul/21 11:17 PM

Assignee

Te-Wei Tsai [ ttsai ]

Russell Owen [ rowen ]

Russell Owen made changes - 12/Jul/21 11:19 PM

Assignee

Russell Owen [ rowen ]

Te-Wei Tsai [ ttsai ]

Hide

Permalink

Te-Wei Tsai added a comment - 13/Jul/21 8:17 PM - edited

Checked the starting point of oscillation of states:

Jul 12 16:10:46 localhost systemd: Starting Stop Read-Ahead Data Collection...

Jul 12 16:10:46 localhost systemd: Started Stop Read-Ahead Data Collection.

Jul 12 16:10:52 localhost journal: LSST Wrapper: new mode= 3.000000

Jul 12 16:10:52 localhost journal: LSST Wrapper: max time in callback =          367132

Jul 12 16:10:52 localhost journal: LSST Wrapper: max time in callback =          399842

Jul 12 16:10:52 localhost journal: LSST Wrapper: max time in callback =          429827

Jul 12 16:10:52 localhost journal: LSST Wrapper: max time in callback =          431242

Jul 12 16:10:54 localhost journal: LSST Wrapper: max time in callback =          446342

Jul 12 16:11:03 localhost journal: LSST Wrapper: max time in callback =          447674

Jul 12 16:11:04 localhost journal: LSST Wrapper: max time in callback =          462521

Jul 12 16:11:32 localhost journal: LSST Wrapper: processed State Cmd: 0x5555, 0x8000, counter=2, parameter1 = 7.000000.

Jul 12 16:11:32 localhost journal: LSST Wrapper: new mode= 0.000000

Jul 12 16:11:32 localhost journal: LSST Wrapper: max time in callback =          498838

Jul 12 16:11:32 localhost journal: LSST Wrapper: new mode= 4.000000

Jul 12 16:11:46 localhost journal: LSST Wrapper: max time in callback =          506773

Jul 12 16:13:56 localhost journal: LSST Wrapper: max time in callback =          592167

Jul 12 16:17:46 localhost journal: LSST Wrapper: Clearing drive faults.

Jul 12 16:17:46 localhost journal: LSST Wrapper: processed State Cmd: 0x5555, 0x8000, counter=3, parameter1 = 6.000000.

Jul 12 16:17:46 localhost journal: LSST Wrapper: STO interlock open

Jul 12 16:17:46 localhost journal: LSST Wrapper: new mode= 0.000000

Jul 12 16:17:46 localhost journal: LSST Wrapper: new mode= 4.000000

Jul 12 16:17:46 localhost journal: LSST Wrapper: new mode= 0.000000

Jul 12 16:17:46 localhost journal: LSST Wrapper: new mode= 4.000000

Jul 12 16:17:46 localhost journal: LSST Wrapper: new mode= 0.000000

Jul 12 16:17:46 localhost journal: LSST Wrapper: new mode= 4.000000

This means the hexapod was in the OfflineState (new mode = 3) first, received the state transition command (0x8000) of EnterControl ( parameter1 = 7), and transitioned to the StandbyState (new mode = 0). After this, there was the interlock error and transitioned to the FaultState (new mode = 4). Then, the ClearError command (parameter1 = 6) is issued. After this, the system began to oscillate between the new modes of StandbyState (0) and FaultState (4).

This behavior is expected as long as the fault is persistent and the ClearError is triggered based on the Simulink state machine (see the transition between the StandbyState and FaultState):

Show

Te-Wei Tsai added a comment - 13/Jul/21 8:17 PM - edited Checked the starting point of oscillation of states: Jul 12 16:10:46 localhost systemd: Starting Stop Read-Ahead Data Collection... Jul 12 16:10:46 localhost systemd: Started Stop Read-Ahead Data Collection. Jul 12 16:10:52 localhost journal: LSST Wrapper: new mode= 3.000000 Jul 12 16:10:52 localhost journal: LSST Wrapper: max time in callback = 367132 Jul 12 16:10:52 localhost journal: LSST Wrapper: max time in callback = 399842 Jul 12 16:10:52 localhost journal: LSST Wrapper: max time in callback = 429827 Jul 12 16:10:52 localhost journal: LSST Wrapper: max time in callback = 431242 Jul 12 16:10:54 localhost journal: LSST Wrapper: max time in callback = 446342 Jul 12 16:11:03 localhost journal: LSST Wrapper: max time in callback = 447674 Jul 12 16:11:04 localhost journal: LSST Wrapper: max time in callback = 462521 Jul 12 16:11:32 localhost journal: LSST Wrapper: processed State Cmd: 0x5555, 0x8000, counter=2, parameter1 = 7.000000. Jul 12 16:11:32 localhost journal: LSST Wrapper: new mode= 0.000000 Jul 12 16:11:32 localhost journal: LSST Wrapper: max time in callback = 498838 Jul 12 16:11:32 localhost journal: LSST Wrapper: new mode= 4.000000 Jul 12 16:11:46 localhost journal: LSST Wrapper: max time in callback = 506773 Jul 12 16:13:56 localhost journal: LSST Wrapper: max time in callback = 592167 Jul 12 16:17:46 localhost journal: LSST Wrapper: Clearing drive faults. Jul 12 16:17:46 localhost journal: LSST Wrapper: processed State Cmd: 0x5555, 0x8000, counter=3, parameter1 = 6.000000. Jul 12 16:17:46 localhost journal: LSST Wrapper: STO interlock open Jul 12 16:17:46 localhost journal: LSST Wrapper: new mode= 0.000000 Jul 12 16:17:46 localhost journal: LSST Wrapper: new mode= 4.000000 Jul 12 16:17:46 localhost journal: LSST Wrapper: new mode= 0.000000 Jul 12 16:17:46 localhost journal: LSST Wrapper: new mode= 4.000000 Jul 12 16:17:46 localhost journal: LSST Wrapper: new mode= 0.000000 Jul 12 16:17:46 localhost journal: LSST Wrapper: new mode= 4.000000 This means the hexapod was in the OfflineState (new mode = 3) first, received the state transition command (0x8000) of EnterControl ( parameter1 = 7), and transitioned to the StandbyState (new mode = 0). After this, there was the interlock error and transitioned to the FaultState (new mode = 4). Then, the ClearError command (parameter1 = 6) is issued. After this, the system began to oscillate between the new modes of StandbyState (0) and FaultState (4). This behavior is expected as long as the fault is persistent and the ClearError is triggered based on the Simulink state machine (see the transition between the StandbyState and FaultState ):

Te-Wei Tsai made changes - 13/Jul/21 8:18 PM

Attachment

stateMachine.png [ 51015 ]

Te-Wei Tsai made changes - 13/Jul/21 9:29 PM

Attachment		dataInEfd.png [ 51016 ]
Attachment		enabledStateWithInvalidCommand.png [ 51017 ]

Hide

Permalink

Te-Wei Tsai added a comment - 13/Jul/21 9:30 PM

Tested the software on summit that the invalid command is executed:

Jul 13 21:15:53 localhost journal: LSST Wrapper: processed State Cmd: 0x5555, 0x8000, counter=0, parameter1 = 0.000000.

Jul 13 21:15:53 localhost journal: LSST Wrapper: processed State Cmd: 0x5555, 0x8000, counter=2, parameter1 = 6.000000.

Jul 13 21:15:53 localhost journal: LSST Wrapper: Clearing drive faults.

Jul 13 21:15:18 localhost journal: LSST Wrapper: max time in callback =          659115

The system can be enabled and the data in EFD:

Show

Te-Wei Tsai added a comment - 13/Jul/21 9:30 PM Tested the software on summit that the invalid command is executed: Jul 13 21:15:53 localhost journal: LSST Wrapper: processed State Cmd: 0x5555, 0x8000, counter=0, parameter1 = 0.000000. Jul 13 21:15:53 localhost journal: LSST Wrapper: processed State Cmd: 0x5555, 0x8000, counter=2, parameter1 = 6.000000. Jul 13 21:15:53 localhost journal: LSST Wrapper: Clearing drive faults. Jul 13 21:15:18 localhost journal: LSST Wrapper: max time in callback = 659115 } The system can be enabled and the data in EFD:

Hide

Permalink

Te-Wei Tsai added a comment - 13/Jul/21 10:23 PM

I realized that I may need to add the sleep time between two commands. Right now, I can stop the oscillation (the low-level controller will try to clear the error for 1 second):

Jul 13 22:07:25 localhost journal: LSST Wrapper: processed State Cmd: 0x5555, 0x8000, counter=0, parameter1 = 0.000000.

(… oscillation)

Jul 13 22:07:25 localhost journal: LSST Wrapper: new mode= 4.000000

Jul 13 22:07:25 localhost journal: LSST Wrapper: new mode= 0.000000

Jul 13 22:07:24 localhost journal: LSST Wrapper: new mode= 4.000000

Jul 13 22:07:24 localhost journal: LSST Wrapper: new mode= 0.000000

Jul 13 22:07:24 localhost journal: LSST Wrapper: new mode= 4.000000

Jul 13 22:07:24 localhost journal: LSST Wrapper: STO interlock open

Jul 13 22:07:24 localhost journal: LSST Wrapper: processed State Cmd: 0x5555, 0x8000, counter=4, parameter1 = 6.000000.

Jul 13 22:07:24 localhost journal: LSST Wrapper: Clearing drive faults

Show

Te-Wei Tsai added a comment - 13/Jul/21 10:23 PM I realized that I may need to add the sleep time between two commands. Right now, I can stop the oscillation (the low-level controller will try to clear the error for 1 second): Jul 13 22:07:25 localhost journal: LSST Wrapper: processed State Cmd: 0x5555, 0x8000, counter=0, parameter1 = 0.000000. (… oscillation) Jul 13 22:07:25 localhost journal: LSST Wrapper: new mode= 4.000000 Jul 13 22:07:25 localhost journal: LSST Wrapper: new mode= 0.000000 Jul 13 22:07:24 localhost journal: LSST Wrapper: new mode= 4.000000 Jul 13 22:07:24 localhost journal: LSST Wrapper: new mode= 0.000000 Jul 13 22:07:24 localhost journal: LSST Wrapper: new mode= 4.000000 Jul 13 22:07:24 localhost journal: LSST Wrapper: STO interlock open Jul 13 22:07:24 localhost journal: LSST Wrapper: processed State Cmd: 0x5555, 0x8000, counter=4, parameter1 = 6.000000. Jul 13 22:07:24 localhost journal: LSST Wrapper: Clearing drive faults

Hide

Permalink

Te-Wei Tsai added a comment - 13/Jul/21 10:30 PM

Please help to review the PR:
https://github.com/lsst-ts/ts_hexapod_controller/pull/20

You had reviewed the commist: 51d3f58 to 1883af9.
The commits: 89094b5 to ab00d5d are based on your previous review.
The commit: b8e0d61 is the new one to solve the state machine problem of Simulink model.

Thanks!

Show

Te-Wei Tsai added a comment - 13/Jul/21 10:30 PM Please help to review the PR: https://github.com/lsst-ts/ts_hexapod_controller/pull/20 You had reviewed the commist: 51d3f58 to 1883af9. The commits: 89094b5 to ab00d5d are based on your previous review. The commit: b8e0d61 is the new one to solve the state machine problem of Simulink model. Thanks!

Te-Wei Tsai made changes - 13/Jul/21 10:30 PM

Status

In Progress [ 3 ]

In Review [ 10004 ]

Hide

Permalink

Te-Wei Tsai added a comment - 15/Jul/21 4:00 AM

This code had been reviewed by Russell in GitHub. Thanks!

Show

Te-Wei Tsai added a comment - 15/Jul/21 4:00 AM This code had been reviewed by Russell in GitHub. Thanks!

Te-Wei Tsai made changes - 15/Jul/21 4:00 AM

Status

In Review [ 10004 ]

Reviewed [ 10101 ]

Te-Wei Tsai made changes - 15/Jul/21 4:00 AM

Resolution		Done [ 10000 ]
Status	Reviewed [ 10101 ]	Done [ 10002 ]

Te-Wei Tsai made changes - 15/Jul/21 2:38 PM

Story Points

Te-Wei Tsai made changes - 16/Jul/21 3:01 PM

Link

This issue relates to ~~DM-30814~~ [ ~~DM-30814~~ ]

People

Assignee:: Te-Wei Tsai

Reporter:: Russell Owen

Reviewers:: Russell Owen

Watchers:: Holger Drass, Petr Kubanek, Russell Owen, Te-Wei Tsai

Votes:: 0 Vote for this issue

Watchers:: 4 Start watching this issue

Dates

Created:: 05/Feb/21 6:23 PM

Updated:: 16/Jul/21 3:01 PM

Resolved:: 15/Jul/21 4:00 AM

Start date:: 24/Jun/21

End date:: 25/Jun/21

Jenkins

No builds found.

Data Management

Details

Description

Attachments

Attachments

Issue Links

Activity

People

Dates

Jenkins