Currently we are working on a scenario which involves simulating a network partition in an akka cluster. While testing it, we found that the actor system in one of the nodes of the downing partition experiences time out during coordinated shutdown.
Below is the log collected from the downing node.
WARN 2023-02-08T07:09:34,518 [FMCluster-akka.actor.default-dispatcher-37] akka.actor.CoordinatedShutdown - Coordinated shutdown phase [actor-system-terminate] timed out after 10000 milliseconds
DEBUG 2023-02-08T07:09:50,999 [FMCluster-akka.actor.default-dispatcher-10] akka.io.TcpIncomingConnection - Closing connection due to IO error java.io.IOException: Connection reset by peer
DEBUG 2023-02-08T07:09:51,021 [FMCluster-akka.actor.default-dispatcher-10] akka.io.TcpIncomingConnection - Closing connection due to IO error java.io.IOException: Connection reset by peer
DEBUG 2023-02-08T07:23:47,247 [FMCluster-akka.actor.default-dispatcher-55] akka.io.TcpOutgoingConnection - Closing connection due to IO error java.io.IOException: Connection timed out
DEBUG 2023-02-08T07:23:47,253 [FMCluster-akka.actor.default-dispatcher-55] akka.remote.artery.ArteryTransport - Outbound connection closed to [akka://FMCluster@214.5.242.152:2551]
WARN 2023-02-08T07:23:47,253 [FMCluster-akka.actor.default-dispatcher-55] akka.stream.Materializer - [outbound connection to [akka://FMCluster@214.5.242.152:2551], control stream] Upstream failed, cause: StreamTcpException: The connection closed with error: Connection timed out
DEBUG 2023-02-08T07:23:47,264 [FMCluster-akka.actor.default-dispatcher-55] akka.stream.impl.io.TcpConnectionStage$TcpStreamLogic - Closing connection from 214.5.242.156:50072 because downstream cancelled stream without failure
DEBUG 2023-02-08T07:23:47,265 [FMCluster-akka.actor.default-dispatcher-55] akka.stream.impl.io.TcpConnectionStage$TcpStreamLogic - Closing connection from 214.5.242.152:38682 because downstream cancelled stream without failure
DEBUG 2023-02-08T07:23:47,267 [FMCluster-akka.actor.default-dispatcher-55] akka.stream.impl.io.TcpConnectionStage$TcpStreamLogic - Closing connection from 214.5.242.156:50070 because downstream cancelled stream without failure
DEBUG 2023-02-08T07:23:47,268 [FMCluster-akka.actor.default-dispatcher-55] akka.stream.impl.io.TcpConnectionStage$TcpStreamLogic - Closing connection from 214.5.242.152:38680 because downstream cancelled stream without failure
DEBUG 2023-02-08T07:23:47,271 [FMCluster-akka.actor.default-dispatcher-55] akka.io.TcpListener - Unbinding endpoint /214.5.242.154:2551
DEBUG 2023-02-08T07:23:47,272 [FMCluster-akka.actor.default-dispatcher-55] akka.io.TcpListener - Unbound endpoint /214.5.242.154:2551, stopping listener
INFO 2023-02-08T07:23:47,284 [FMCluster-akka.actor.default-dispatcher-55] akka.remote.RemoteActorRefProvider$RemotingTerminator - Remoting shut down.
The Coordinated shutdown task experiences a time-out while closing artery TCP remoting sockets and hangs for a longtime. The system is kept idle for some time and ~860 seconds later the Actor system resumes the termination.
During this hang duration we checked netstat and found that some TCP connections are in FIN_WAIT_1 state.
After ~860 seconds, the log statements indicate that the actor system termination is resumed and TCP connections in FIN_WAIT1 state are closed after a time out.
This issue looks more similar to the problem discussed in the below link.
Actor System takes 15 Minutes to terminate completely For Artery Remoting · Issue #30326 · akka/akka · GitHub
We couldn’t figure out the reason for such happenings. Can anyone shed some light here?