Prevent akka remote to reassociate after association error


(Martin Grotzke) #1

We are running a distributed system where one service (multiple instances) is running inside a kubernetes cluster (k8s), and other nodes are running in other data centers outside the k8s cluster. Lets say node A runs in k8s, and node B runs outside k8s. Both are connected via akka remoting (netty tcp, akka 2.5.14), where A establishes the connection to B. Once there’s a connection error, node B also attempts to reestablish the connection/association. This however fails, because A is not routable from B (due to the k8s pod/service configuration). Reproducting this issue (by dropping outbound packets from B to A) shows that while reestabilishing the connection from B to A fails, A can reestablish the connection as soon as packets are no longer dropped.

My question: is it possible to prevent B from reestablishing the connection to A? I’ve stumbled over “akka.remote.adapters” (TransportAdapterProvider), maybe this allows to do this?

I’m well aware that akka remoting not a client-server but a peer-to-peer technology, so another solution would be to switch to e.g. http as transport. Switching the transport however with multiple DCs involved is not such a simple thing, therefore I’m looking for a solution with less impact regarding the infrastructure.


(Patrik Nordwall) #2

As you have noted Akka Remoting requires peer-to-peer connectivity. I recommend against circumventing that prerequisite.

I wouldn’t base a solution on that. That should probably never have been public API, if it is. Note that we will also phase out the classic remoting in favor of Artery (TCP).

I would recommend Akka gRPC or Akka HTTP for client-server setups, or configure k8 for access in both directions.


(Martin Grotzke) #3

Thanks for your suggestion, @patriknw!

We’ve now replaced akka remoting with http, because a) this better represents the communication flow and is a simpler model in that regard and b) because k8s does not “natively” / easily supports point to point connections to pods from the outside.