While testing the ATPtg with the ATMCS (DM-19541) I noticed that, after leaving tracking for a while, commands against ATMCS would take a long time to be acknowledged. After debugging it for some time I figured that the issue is a combination of how acknowledgements are handle by SAL and SalObj. Basically, for each component there is a single ackcmd topic. So, all commands for a single component share the same topic for acknowledging a command. So, assume there are three components C1, C2 and C3.

C1 has a remote* for C2 and C3
C2 is a controller* that has a remote for C3
C3 is a controller with no remotes.

when C1 sends a command to C2, C2 will start to stream commands to C3. The acknowledgment sent by C3 for each command it receives is queued by C1. If C1 need to send a command to C3, it will have to check all the queued acks before arriving at the intended one (probably the last one in the queue).

To fix this C1 could be continuously checking for acks on salobj level. The dds version of salobj is already implemented to do so. We should consider doing the same for salpy version of salobj.

*remote = sends commands and listens for telemetry and events.
*controller = listen for commands and sends telemetry and events