Odd behavior with redundant/standby scanner

[troglobit]

Hi! My end-customer have started their CIP acceptance test phase now
and found an interesting issue I cannot wrap my head around.

They have two active scanners connecting to my adapter. One acting as
master and one as an active standby (slave):

• The master produces assembly 101 (unicast) and consumes 100 (multicast)
• The standby produce heartbeat 254 (unicast) and consume 100 (multicast)

When they pull the network cable to the master, to verify the standby
scanner continues to receive multicast, they instead notice the standby
scanner being disconnected. It tries to reconnect but there is no T->O
communication anymore, only the O->T and due to this the standby scanner
gives up and retries again, and again ... In fact, the standby scanner
never recovers in this case until the master scanner is reconnected (!)
and reopens its connection. (It's all multicast though so I'm not sure
everything is fine and dandy in the adapter, but it looks OK from the
outside since the standby scanner receives T->O data.)

If they instead of pulling the cable just terminate the CIP connection
of the master, the standby scanner is happy and receives multicast data
as it should. (Terminate, meaning proper close and ENIP unregister.)

In the log I found an interesting difference. In CloseIoConnection():

2022-08-30T13:57:59 [NFO]: Exclusive Owner or Input Only connection closed - Instance type :2
2022-08-30T13:57:59 [NFO]: Transfer socket ownership

Whereas in HandleIoConnectionTimeOut() there is no such handling at
all. This function comes directly after the former and they are very
similar, so I played around a bit with the latter:

void HandleIoConnectionTimeOut(CipConnectionObject *connection_object) {
#if 1
  CloseIoConnection(connection_object);
#else
  ...
#endif
}

With this ugly hack I get at least similar behavior, i.e., the standby
scanner continues receiving data when pulling the master's cable.

This is by no means a fix, since everything else breaks, e.g., the
standby scanner can no longer be stopped and restarted. It is however
an interesting observation so I wanted to hear your input on this.

[MartinMelikMerkumians]

With standby scanner I assume it is a second scanner which opened an input only connection?

In that case I am a bit confused, as I implemented the appropriate behavior some time ago. In case of a timeout of the EO connection, the EO connections should get closed and checked if there is an IO connection. If yes, all IO and LO connections should still work, if it was the last EO (or IO) connection then the LOs need to be terminated as well.

I am rather sure this was also tested at PlugFest in 2019, also by disconnecting the device and not by closing the connection.

[MartinMelikMerkumians]

oh, I think the case kConnectionObjectInstanceTypeIOExclusiveOwner is missing the ownership transfer part, It closes all connection if I am not mistaken

[MartinMelikMerkumians]

Reading through the specification, the code and you scenario, I am rather sure the Input Only connection gets terminated too.
As the connection has "timeouted", it is probably closed or marked as timeouted. Without a Wireshark trace I am not sure if the second scanner tries to reconnect (which should be theoretically successful I guess).

However, when the first scanner reconnect, it can reuse the connection and the other connection opens again too, as it seems

[troglobit]

Yes, correct, the second scanner opens an input-only connection.

Yeah, I'm pretty sure there's a missing ownership transfer. In fact, that was my theory which I've been playing around with since yesterday. I have a working prototype now, that both hands over the connection properly on timeout, and handles the case when reconnecting either the master or standby scanner.

I'm working on cleaning up the code now, but I expect to have at least an RFC patch ready for your review as a pull request, very soon. I'll be including a few other minor fixups as well, that I noticed while troubleshooting, but the majority of my changes are in cipioconnection.c. I'm not at all 100% that the changes cover all cases in the standard, but traffic flows smoothly and is never interrupted -- and there are no socket leaks either as far as I can see right now.

[troglobit]

First RFC pull request available as PR #394