We are running a distributed system where one service (multiple instances) is running inside a kubernetes cluster (k8s), and other nodes are running in other data centers outside the k8s cluster. Lets say node A runs in k8s, and node B runs outside k8s. Both are connected via akka remoting (netty tcp, akka 2.5.14), where A establishes the connection to B. Once there’s a connection error, node B also attempts to reestablish the connection/association. This however fails, because A is not routable from B (due to the k8s pod/service configuration). Reproducting this issue (by dropping outbound packets from B to A) shows that while reestabilishing the connection from B to A fails, A can reestablish the connection as soon as packets are no longer dropped.
My question: is it possible to prevent B from reestablishing the connection to A? I’ve stumbled over “akka.remote.adapters” (TransportAdapterProvider), maybe this allows to do this?
I’m well aware that akka remoting not a client-server but a peer-to-peer technology, so another solution would be to switch to e.g. http as transport. Switching the transport however with multiple DCs involved is not such a simple thing, therefore I’m looking for a solution with less impact regarding the infrastructure.