Lagom 1.4.11 - Association failed Caused by: [The remote system explicitly disassociated (reason unknown)

Hello all,

We’ve upgraded our services to Lagom 1.4.11, and using the latest sbt-reactive-app 1.7.1.

One small issue we found today is that while RP is generating the kubernetes deployment YAML files assuming that play is listening on port 10000, the latest reactive cli is using the port 9000 as a default. However, this is not a major issue as you can quite easily override the http port in your application.conf, which we did.

After upgrading to the 1.4.11, none of our deployments were able to form the cluster sharding and I suspect this error is the issue:

Association with remote system [akka.tcp://application@10-8-0-186.coolapp-testing.pod.cluster.local:2552] has failed, address is now gated for [5000] ms. Reason: [Association failed with [akka.tcp://application@10-8-0-186.coolapp-testing.pod.cluster.local:2552]] Caused by: [The remote system explicitly disassociated (reason unknown).]

We still couldn’t really figure out what is causing this error. And we are still investigating this issue at the moment but sharing here first anyways, just in case someone has the clue for it.

Thanks,

The full log message is:

2019-02-13T04:25:00.675Z [e[37minfoe[0m] play.core.server.AkkaHttpServer [] - Listening for HTTP on /127.0.0.1:10000
2019-02-13T04:25:09.176Z [e[33mwarne[0m] akka.cluster.Cluster(akka://application) [sourceThread=application-akka.actor.default-dispatcher-5, akkaTimestamp=04:25:09.176UTC, akkaSource=akka.cluster.Cluster(akka://application), sourceActorSystem=application] - Cluster Node [akka.tcp://application@10-8-1-7.coolapp-testing.pod.cluster.local:2552] - Couldn't join seed nodes after [2] attempts, will try again. seed-nodes=[akka.tcp://application@10-8-0-186.coolapp-testing.pod.cluster.local:2552]
2019-02-13T04:25:09.347Z [e[33mwarne[0m] akka.remote.ReliableDeliverySupervisor [sourceThread=application-akka.remote.default-remote-dispatcher-23, akkaTimestamp=04:25:09.346UTC, akkaSource=akka.tcp://application@10-8-1-7.coolapp-testing.pod.cluster.local:2552/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2Fapplication%4010-8-0-186.coolapp-testing.pod.cluster.local%3A2552-0, sourceActorSystem=application] - Association with remote system [akka.tcp://application@10-8-0-186.coolapp-testing.pod.cluster.local:2552] has failed, address is now gated for [5000] ms. Reason: [Association failed with [akka.tcp://application@10-8-0-186.coolapp-testing.pod.cluster.local:2552]] Caused by: [The remote system explicitly disassociated (reason unknown).]
2019-02-13T04:25:14.225Z [e[33mwarne[0m] akka.cluster.Cluster(akka://application) [sourceThread=application-akka.actor.default-dispatcher-5, akkaTimestamp=04:25:14.224UTC, akkaSource=akka.cluster.Cluster(akka://application), sourceActorSystem=application] - Cluster Node [akka.tcp://application@10-8-1-7.coolapp-testing.pod.cluster.local:2552] - Couldn't join seed nodes after [3] attempts, will try again. seed-nodes=[akka.tcp://application@10-8-0-186.coolapp-testing.pod.cluster.local:2552]
2019-02-13T04:25:19.275Z [e[33mwarne[0m] akka.cluster.Cluster(akka://application) [sourceThread=application-akka.actor.default-dispatcher-15, akkaTimestamp=04:25:19.275UTC, akkaSource=akka.cluster.Cluster(akka://application), sourceActorSystem=application] - Cluster Node [akka.tcp://application@10-8-1-7.coolapp-testing.pod.cluster.local:2552] - Couldn't join seed nodes after [4] attempts, will try again. seed-nodes=[akka.tcp://application@10-8-0-186.coolapp-testing.pod.cluster.local:2552]
2019-02-13T04:25:19.432Z [e[33mwarne[0m] akka.remote.ReliableDeliverySupervisor [sourceThread=application-akka.remote.default-remote-dispatcher-24, akkaTimestamp=04:25:19.431UTC, akkaSource=akka.tcp://application@10-8-1-7.coolapp-testing.pod.cluster.local:2552/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2Fapplication%4010-8-0-186.coolapp-testing.pod.cluster.local%3A2552-0, sourceActorSystem=application] - Association with remote system [akka.tcp://application@10-8-0-186.coolapp-testing.pod.cluster.local:2552] has failed, address is now gated for [5000] ms. Reason: [Association failed with [akka.tcp://application@10-8-0-186.coolapp-testing.pod.cluster.local:2552]] Caused by: [The remote system explicitly disassociated (reason unknown).]
2019-02-13T04:25:24.305Z [e[33mwarne[0m] akka.cluster.Cluster(akka://application) [sourceThread=application-akka.actor.default-dispatcher-2, akkaTimestamp=04:25:24.304UTC, akkaSource=akka.cluster.Cluster(akka://application), sourceActorSystem=application] - Cluster Node [akka.tcp://application@10-8-1-7.coolapp-testing.pod.cluster.local:2552] - Couldn't join seed nodes after [5] attempts, will try again. seed-nodes=[akka.tcp://application@10-8-0-186.coolapp-testing.pod.cluster.local:2552]
2019-02-13T04:25:29.335Z [e[33mwarne[0m] akka.cluster.Cluster(akka://application) [sourceThread=application-akka.actor.default-dispatcher-21, akkaTimestamp=04:25:29.334UTC, akkaSource=akka.cluster.Cluster(akka://application), sourceActorSystem=application] - Cluster Node [akka.tcp://application@10-8-1-7.coolapp-testing.pod.cluster.local:2552] - Couldn't join seed nodes after [6] attempts, will try again. seed-nodes=[akka.tcp://application@10-8-0-186.coolapp-testing.pod.cluster.local:2552]
2019-02-13T04:25:29.514Z [e[33mwarne[0m] akka.remote.ReliableDeliverySupervisor [sourceThread=application-akka.remote.default-remote-dispatcher-23, akkaTimestamp=04:25:29.513UTC, akkaSource=akka.tcp://application@10-8-1-7.coolapp-testing.pod.cluster.local:2552/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2Fapplication%4010-8-0-186.coolapp-testing.pod.cluster.local%3A2552-0, sourceActorSystem=application] - Association with remote system [akka.tcp://application@10-8-0-186.coolapp-testing.pod.cluster.local:2552] has failed, address is now gated for [5000] ms. Reason: [Association failed with [akka.tcp://application@10-8-0-186.coolapp-testing.pod.cluster.local:2552]] Caused by: [The remote system explicitly disassociated (reason unknown).]
2019-02-13T04:25:34.354Z [e[33mwarne[0m] akka.cluster.Cluster(akka://application) [sourceThread=application-akka.actor.default-dispatcher-2, akkaTimestamp=04:25:34.354UTC, akkaSource=akka.cluster.Cluster(akka://application), sourceActorSystem=application] - Cluster Node [akka.tcp://application@10-8-1-7.coolapp-testing.pod.cluster.local:2552] - Couldn't join seed nodes after [7] attempts, will try again. seed-nodes=[akka.tcp://application@10-8-0-186.coolapp-testing.pod.cluster.local:2552]
2019-02-13T04:25:39.255Z [e[33mwarne[0m] akka.cluster.Cluster(akka://application) [sourceThread=application-akka.actor.default-dispatcher-15, akkaTimestamp=04:25:39.254UTC, akkaSource=akka.cluster.Cluster(akka://application), sourceActorSystem=application] - Cluster Node [akka.tcp://application@10-8-1-7.coolapp-testing.pod.cluster.local:2552] - Joining of seed-nodes [akka.tcp://application@10-8-0-186.coolapp-testing.pod.cluster.local:2552] was unsuccessful after configured shutdown-after-unsuccessful-join-seed-nodes [40000 milliseconds]. Running CoordinatedShutdown.
2019-02-13T04:25:39.281Z [e[37minfoe[0m] akka.cluster.singleton.ClusterSingletonManager [sourceThread=application-akka.actor.default-dispatcher-20, akkaTimestamp=04:25:39.281UTC, akkaSource=akka.tcp://application@10-8-1-7.coolapp-testing.pod.cluster.local:2552/system/sharding/AdminEntityCoordinator, sourceActorSystem=application] - Exited [akka.tcp://application@10-8-1-7.coolapp-testing.pod.cluster.local:2552]
2019-02-13T04:25:39.284Z [e[37minfoe[0m] akka.cluster.singleton.ClusterSingletonManager [sourceThread=application-akka.actor.default-dispatcher-3, akkaTimestamp=04:25:39.282UTC, akkaSource=akka.tcp://application@10-8-1-7.coolapp-testing.pod.cluster.local:2552/user/cassandraOffsetStorePrepare-singleton, sourceActorSystem=application] - Exited [akka.tcp://application@10-8-1-7.coolapp-testing.pod.cluster.local:2552]
2019-02-13T04:25:39.284Z [e[37minfoe[0m] akka.cluster.Cluster(akka://application) [sourceThread=application-akka.actor.default-dispatcher-20, akkaTimestamp=04:25:39.283UTC, akkaSource=akka.cluster.Cluster(akka://application), sourceActorSystem=application] - Cluster Node [akka.tcp://application@10-8-1-7.coolapp-testing.pod.cluster.local:2552] - Exiting completed
2019-02-13T04:25:39.289Z [e[37minfoe[0m] akka.cluster.Cluster(akka://application) [sourceThread=application-akka.actor.default-dispatcher-20, akkaTimestamp=04:25:39.287UTC, akkaSource=akka.cluster.Cluster(akka://application), sourceActorSystem=application] - Cluster Node [akka.tcp://application@10-8-1-7.coolapp-testing.pod.cluster.local:2552] - Shutting down...
2019-02-13T04:25:39.292Z [e[37minfoe[0m] akka.cluster.Cluster(akka://application) [sourceThread=application-akka.actor.default-dispatcher-20, akkaTimestamp=04:25:39.289UTC, akkaSource=akka.cluster.Cluster(akka://application), sourceActorSystem=application] - Cluster Node [akka.tcp://application@10-8-1-7.coolapp-testing.pod.cluster.local:2552] - Successfully shut down

I found some additional error logs after lowering the Akka log level to DEBUG:

At the beginning of the clustering process, it seems like bootstrapCoordinator runs into this error first:

Probing [http://10-8-2-184.coolapp-testing.pod.cluster.local:8558/bootstrap/seed-nodes] failed due to: Unsupported Content-Type, supported: application/json


2019-02-13T06:12:19.488Z [e[37minfoe[0m] akka.management.AkkaManagement [sourceThread=application-akka.actor.default-dispatcher-2, akkaTimestamp=06:12:19.487UTC, akkaSource=AkkaManagement(akka://application), sourceActorSystem=application] - Bound Akka Management (HTTP) endpoint to: 127.0.0.1:8558
2019-02-13T06:12:20.815Z [e[37minfoe[0m] akka.management.cluster.bootstrap.internal.BootstrapCoordinator [sourceThread=application-akka.actor.default-dispatcher-16, akkaTimestamp=06:12:20.813UTC, akkaSource=akka.tcp://application@10-8-3-223.coolapp-testing.pod.cluster.local:2552/system/bootstrapCoordinator, sourceActorSystem=application] - Located service members based on: [Lookup(admin,Some(management),Some(tcp))]: [ResolvedTarget(10-8-2-184.coolapp-testing.pod.cluster.local,Some(8558),Some(/10.8.2.184)), ResolvedTarget(10-8-1-9.coolapp-testing.pod.cluster.local,Some(8558),Some(/10.8.1.9)), ResolvedTarget(10-8-3-223.coolapp-testing.pod.cluster.local,Some(8558),Some(/10.8.3.223))]
2019-02-13T06:12:20.853Z [e[37minfoe[0m] akka.management.cluster.bootstrap.internal.BootstrapCoordinator [sourceThread=application-akka.actor.default-dispatcher-16, akkaSource=akka.tcp://application@10-8-3-223.coolapp-testing.pod.cluster.local:2552/system/bootstrapCoordinator, sourceActorSystem=application, akkaTimestamp=06:12:20.852UTC] - Located service members based on: [Lookup(admin,Some(management),Some(tcp))]: [ResolvedTarget(10-8-2-184.coolapp-testing.pod.cluster.local,Some(8558),Some(/10.8.2.184)), ResolvedTarget(10-8-1-9.coolapp-testing.pod.cluster.local,Some(8558),Some(/10.8.1.9)), ResolvedTarget(10-8-3-223.coolapp-testing.pod.cluster.local,Some(8558),Some(/10.8.3.223))]
2019-02-13T06:12:21.265Z [e[33mwarne[0m] akka.management.cluster.bootstrap.internal.HttpContactPointBootstrap [sourceThread=application-akka.actor.default-dispatcher-2, akkaTimestamp=06:12:21.257UTC, akkaSource=akka.tcp://application@10-8-3-223.coolapp-testing.pod.cluster.local:2552/system/bootstrapCoordinator/contactPointProbe-10-8-2-184.coolapp-testing.pod.cluster.local-8558, sourceActorSystem=application] - Probing [http://10-8-2-184.coolapp-testing.pod.cluster.local:8558/bootstrap/seed-nodes] failed due to: Unsupported Content-Type, supported: application/json
2019-02-13T06:12:21.266Z [e[37minfoe[0m] akka.management.cluster.bootstrap.internal.BootstrapCoordinator [sourceThread=application-akka.actor.default-dispatcher-16, akkaTimestamp=06:12:21.264UTC, akkaSource=akka.tcp://application@10-8-3-223.coolapp-testing.pod.cluster.local:2552/system/bootstrapCoordinator, sourceActorSystem=application] - Contact point [akka.tcp://application@10-8-1-9.coolapp-testing.pod.cluster.local:2552] returned [1] seed-nodes [akka.tcp://application@10-8-1-9.coolapp-testing.pod.cluster.local:2552]
2019-02-13T06:12:21.320Z [e[37minfoe[0m] akka.management.cluster.bootstrap.internal.BootstrapCoordinator [sourceThread=application-akka.actor.default-dispatcher-16, akkaTimestamp=06:12:21.320UTC, akkaSource=akka.tcp://application@10-8-3-223.coolapp-testing.pod.cluster.local:2552/system/bootstrapCoordinator, sourceActorSystem=application] - Joining [akka.tcp://application@10-8-3-223.coolapp-testing.pod.cluster.local:2552] to existing cluster [akka.tcp://application@10-8-1-9.coolapp-testing.pod.cluster.local:2552]
2019-02-13T06:12:21.717Z [e[37minfoe[0m] akka.management.cluster.bootstrap.contactpoint.HttpClusterBootstrapRoutes [sourceThread=application-akka.actor.default-dispatcher-21, akkaTimestamp=06:12:21.717UTC, akkaSource=HttpClusterBootstrapRoutes(akka://application), sourceActorSystem=application] - Bootstrap request from 127.0.0.1:47636: Contact Point returning 0 seed-nodes ([Set()])

@TimMoore sorry for tagging again, but we’ve tried this again in the fresh environment but still no luck. Is there any known compatibility issue between the Lagom 1.4.11 version with the newest Reactive CLI?

Not that I know of. I’ll ask around.

I don’t know what is causing this.
In the end it can’t connect to akka.tcp://application@10-8-0-186.coolapp-testing.pod.cluster.local:2552.
Is that the expected hostname and port?

@patriknw That’s the correct hostname and port in our kubernetes cluster.
Probing it with nc -vz showed that the port is open and listening.

Is there anything we can do like e.g. curl/netcat to probe it further?

For more context, this is why our hostname and pod looks a little… odd

We haven’t done a lot of testing with Istio yet, and it’s quite likely that it is interfering with communication between pods. When you did the netcat testing, were you running nc from the same pod, or the one that is trying to connect to it?

We nc from origin to destination. This is the sample command

nc -vz 10-8-9-53.coolapp-testing.pod.cluster.local 2552

and we get

10-8-9-53.coolapp-testing.pod.cluster.local (10.8.9.53:2552) open

in response.

So the hypothesis is that the destination pod didn’t actually receive cluster formation request from the origin pod, is that correct? We can deploy it again and monitor istio logs, but iirc, istio-proxy showed connection requests on port 2552 from origin pod.

Are there relevant log messages on the destination pod?

Went back and dig abit, here are the logs

Source (follower):

akka.management.cluster.bootstrap.internal.BootstrapCoordinator [sourceThread=application-akka.actor.default-dispatcher-19, akkaTimestamp=07:01:39.944UTC, akkaSource=akka.tcp://application@10-8-2-188.coolapp-testing.pod.cluster.local:2552/system/bootstrapCoordinator, sourceActorSystem=application] - Looking up [Lookup(admin,Some(management),Some(tcp))]

akka.management.cluster.bootstrap.internal.BootstrapCoordinator [sourceThread=application-akka.actor.default-dispatcher-15, akkaTimestamp=07:01:39.974UTC, akkaSource=akka.tcp://application@10-8-2-188.coolapp-testing.pod.cluster.local:2552/system/bootstrapCoordinator, sourceActorSystem=application] - Located service members based on: [Lookup(admin,Some(management),Some(tcp))]: [ResolvedTarget(10-8-2-188.coolapp-testing.pod.cluster.local,Some(8558),Some(/10.8.2.188)), ResolvedTarget(10-8-0-188.coolapp-testing.pod.cluster.local,Some(8558),Some(/10.8.0.188)), ResolvedTarget(10-8-1-13.coolapp-testing.pod.cluster.local,Some(8558),Some(/10.8.1.13))]

akka.management.cluster.bootstrap.internal.BootstrapCoordinator [sourceThread=application-akka.actor.default-dispatcher-19, akkaTimestamp=07:01:40.134UTC, akkaSource=akka.tcp://application@10-8-2-188.coolapp-testing.pod.cluster.local:2552/system/bootstrapCoordinator, sourceActorSystem=application] - Contact point [akka.tcp://application@10-8-2-188.coolapp-testing.pod.cluster.local:2552] returned [0] seed-nodes []

akka.management.cluster.bootstrap.internal.BootstrapCoordinator [sourceThread=application-akka.actor.default-dispatcher-4, akkaTimestamp=07:01:40.248UTC, akkaSource=akka.tcp://application@10-8-2-188.coolapp-testing.pod.cluster.local:2552/system/bootstrapCoordinator, sourceActorSystem=application] - Joining [akka.tcp://application@10-8-2-188.coolapp-testing.pod.cluster.local:2552] to existing cluster [akka.tcp://application@10-8-0-188.coolapp-testing.pod.cluster.local:2552]

akka.management.cluster.bootstrap.internal.BootstrapCoordinator [sourceThread=application-akka.actor.default-dispatcher-4, akkaTimestamp=07:01:40.245UTC, akkaSource=akka.tcp://application@10-8-2-188.coolapp-testing.pod.cluster.local:2552/system/bootstrapCoordinator, sourceActorSystem=application] - Contact point [akka.tcp://application@10-8-0-188.coolapp-testing.pod.cluster.local:2552] returned [1] seed-nodes [akka.tcp://application@10-8-0-188.coolapp-testing.pod.cluster.local:2552]

akka.remote.ReliableDeliverySupervisor [sourceThread=application-akka.remote.default-remote-dispatcher-31, akkaTimestamp=07:01:40.718UTC, akkaSource=akka.tcp://application@10-8-2-188.coolapp-testing.pod.cluster.local:2552/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2Fapplication%4010-8-0-188.coolapp-testing.pod.cluster.local%3A2552-0, sourceActorSystem=application] - Association with remote system [akka.tcp://application@10-8-0-188.coolapp-testing.pod.cluster.local:2552] has failed, address is now gated for [5000] ms. Reason: [Association failed with [akka.tcp://application@10-8-0-188.coolapp-testing.pod.cluster.local:2552]] Caused by: [The remote system explicitly disassociated (reason unknown).]

akka.cluster.Cluster(akka://application) [sourceThread=application-akka.actor.default-dispatcher-5, akkaTimestamp=07:01:50.435UTC, akkaSource=akka.cluster.Cluster(akka://application), sourceActorSystem=application] - Cluster Node [akka.tcp://application@10-8-2-188.coolapp-testing.pod.cluster.local:2552] - Couldn't join seed nodes after [2] attempts, will try again. seed-nodes=[akka.tcp://application@10-8-0-188.coolapp-testing.pod.cluster.local:2552]

akka.remote.ReliableDeliverySupervisor [sourceThread=application-akka.remote.default-remote-dispatcher-26, akkaTimestamp=07:01:50.637UTC, akkaSource=akka.tcp://application@10-8-2-188.coolapp-testing.pod.cluster.local:2552/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2Fapplication%4010-8-0-188.coolapp-testing.pod.cluster.local%3A2552-0, sourceActorSystem=application] - Association with remote system [akka.tcp://application@10-8-0-188.coolapp-testing.pod.cluster.local:2552] has failed, address is now gated for [5000] ms. Reason: [Association failed with [akka.tcp://application@10-8-0-188.coolapp-testing.pod.cluster.local:2552]] Caused by: [The remote system explicitly disassociated (reason unknown).]

akka.cluster.Cluster(akka://application) [sourceThread=application-akka.actor.default-dispatcher-5, akkaTimestamp=07:01:55.495UTC, akkaSource=akka.cluster.Cluster(akka://application), sourceActorSystem=application] - Cluster Node [akka.tcp://application@10-8-2-188.coolapp-testing.pod.cluster.local:2552] - Couldn't join seed nodes after [3] attempts, will try again. seed-nodes=[akka.tcp://application@10-8-0-188.coolapp-testing.pod.cluster.local:2552]

Destination (Leader):

akka.management.cluster.bootstrap.internal.BootstrapCoordinator [sourceThread=application-akka.actor.default-dispatcher-17, akkaTimestamp=07:01:38.661UTC, akkaSource=akka.tcp://application@10-8-0-188.coolapp-testing.pod.cluster.local:2552/system/bootstrapCoordinator, sourceActorSystem=application] - Looking up [Lookup(admin,Some(management),Some(tcp))]

akka.management.cluster.bootstrap.internal.BootstrapCoordinator [sourceThread=application-akka.actor.default-dispatcher-4, akkaTimestamp=07:01:38.738UTC, akkaSource=akka.tcp://application@10-8-0-188.coolapp-testing.pod.cluster.local:2552/system/bootstrapCoordinator, sourceActorSystem=application] - Contact point [akka.tcp://application@10-8-1-13.coolapp-testing.pod.cluster.local:2552] returned [0] seed-nodes []

akka.management.cluster.bootstrap.internal.BootstrapCoordinator [sourceThread=application-akka.actor.default-dispatcher-4, akkaTimestamp=07:01:38.761UTC, akkaSource=akka.tcp://application@10-8-0-188.coolapp-testing.pod.cluster.local:2552/system/bootstrapCoordinator, sourceActorSystem=application] - Located service members based on: [Lookup(admin,Some(management),Some(tcp))]: [ResolvedTarget(10-8-2-188.coolapp-testing.pod.cluster.local,Some(8558),Some(/10.8.2.188)), ResolvedTarget(10-8-0-188.coolapp-testing.pod.cluster.local,Some(8558),Some(/10.8.0.188)), ResolvedTarget(10-8-1-13.coolapp-testing.pod.cluster.local,Some(8558),Some(/10.8.1.13))]

akka.management.cluster.bootstrap.internal.BootstrapCoordinator [sourceThread=application-akka.actor.default-dispatcher-20, akkaTimestamp=07:01:38.774UTC, akkaSource=akka.tcp://application@10-8-0-188.coolapp-testing.pod.cluster.local:2552/system/bootstrapCoordinator, sourceActorSystem=application] - Contact point [akka.tcp://application@10-8-2-188.coolapp-testing.pod.cluster.local:2552] returned [0] seed-nodes []

akka.management.cluster.bootstrap.internal.BootstrapCoordinator [sourceThread=application-akka.actor.default-dispatcher-20, akkaTimestamp=07:01:39.303UTC, akkaSource=akka.tcp://application@10-8-0-188.coolapp-testing.pod.cluster.local:2552/system/bootstrapCoordinator, sourceActorSystem=application] - Initiating new cluster, self-joining [akka.tcp://application@10-8-0-188.coolapp-testing.pod.cluster.local:2552]. Other nodes are expected to locate this cluster via continued contact-point probing.

akka.cluster.Cluster(akka://application) [sourceThread=application-akka.actor.default-dispatcher-3, akkaTimestamp=07:01:39.336UTC, akkaSource=akka.cluster.Cluster(akka://application), sourceActorSystem=application] - Cluster Node [akka.tcp://application@10-8-0-188.coolapp-testing.pod.cluster.local:2552] - Node [akka.tcp://application@10-8-0-188.coolapp-testing.pod.cluster.local:2552] is JOINING itself (with roles [dc-default]) and forming new cluster

akka.cluster.Cluster(akka://application) [sourceThread=application-akka.actor.default-dispatcher-3, akkaTimestamp=07:01:39.349UTC, akkaSource=akka.cluster.Cluster(akka://application), sourceActorSystem=application] - Cluster Node [akka.tcp://application@10-8-0-188.coolapp-testing.pod.cluster.local:2552] - is the new leader among reachable nodes (more leaders may exist)

// After waiting for 7 mins, failing to form cluster

akka.cluster.Cluster(akka://application) [sourceThread=application-akka.actor.default-dispatcher-20, akkaTimestamp=07:08:40.823UTC, akkaSource=akka.cluster.Cluster(akka://application), sourceActorSystem=application] - Cluster Node [akka.tcp://application@10-8-0-188.coolapp-testing.pod.cluster.local:2552] - Marked address [akka.tcp://application@10-8-0-188.coolapp-testing.pod.cluster.local:2552] as [Leaving]

akka.cluster.singleton.ClusterSingletonManager [sourceThread=application-akka.actor.default-dispatcher-16, akkaTimestamp=07:08:40.829UTC, akkaSource=akka.tcp://application@10-8-0-188.coolapp-testing.pod.cluster.local:2552/user/cassandraOffsetStorePrepare-singleton, sourceActorSystem=application] - Exited [akka.tcp://application@10-8-0-188.coolapp-testing.pod.cluster.local:2552]

akka.cluster.singleton.ClusterSingletonManager [sourceThread=application-akka.actor.default-dispatcher-4, akkaTimestamp=07:08:40.829UTC, akkaSource=akka.tcp://application@10-8-0-188.coolapp-testing.pod.cluster.local:2552/system/sharding/AdminEntityCoordinator, sourceActorSystem=application] - Exited [akka.tcp://application@10-8-0-188.coolapp-testing.pod.cluster.local:2552]

akka.cluster.Cluster(akka://application) [sourceThread=application-akka.actor.default-dispatcher-15, akkaTimestamp=07:08:40.836UTC, akkaSource=akka.cluster.Cluster(akka://application), sourceActorSystem=application] - Cluster Node [akka.tcp://application@10-8-0-188.coolapp-testing.pod.cluster.local:2552] - Leader is moving node [akka.tcp://application@10-8-0-188.coolapp-testing.pod.cluster.local:2552] to [Exiting]

akka.cluster.Cluster(akka://application) [sourceThread=application-akka.actor.default-dispatcher-15, akkaTimestamp=07:08:40.845UTC, akkaSource=akka.cluster.Cluster(akka://application), sourceActorSystem=application] - Cluster Node [akka.tcp://application@10-8-0-188.coolapp-testing.pod.cluster.local:2552] - Exiting completed

@TimMoore just wondering if Lightbend team had a chance to take a look at this issue. I am curious to find out if the 1.4.11 deployment worked without any problems in the other team’s projects.If it worked, how their environments are different to ours…

@lejoow we don’t know what’s wrong. There are other Lagom users who have deployed 1.4.11 successfully. As I mentioned above, we haven’t done much testing with Istio, so I think it’s likely that it’s involved. If there are no messages on application@10-8-0-188.coolapp-testing.pod.cluster.local:2552 showing other nodes trying to join, then I think the best explanation is that the connection isn’t established successfully.

@lejoow Hi Joo. We are seeing a similar issue where we see Association with remote system ... has failed caused by The remote system explicitly disassociated (reason unknown). I see this is a fairly old discussion, but did you ever find a solution or cause for this issue? Do you happen to recall which version of Lagom you uprgraded from to v 1.4.11? The changes to our classpath include the following changes:

Working clustered akka system:
commons-codec.commons-codec-1.10.jar
org.apache.httpcomponents.httpclient-4.5.5.jar
org.apache.httpcomponents.httpcore-4.4.9.jar
com.fasterxml.jackson.core.jackson-annotations-2.8.0.jar
com.fasterxml.jackson.core.jackson-databind-2.8.4.jar
com.fasterxml.jackson.dataformat.jackson-dataformat-cbor-2.8.10.jar

Upgrades to these libs in breaking clustered akka system:
commons-codec.commons-codec-1.11.jar
org.apache.httpcomponents.httpclient-4.5.9.jar
org.apache.httpcomponents.httpcore-4.4.11.jar
com.fasterxml.jackson.core.jackson-annotations-2.9.8.jar
com.fasterxml.jackson.core.jackson-databind-2.9.8.jar
com.fasterxml.jackson.dataformat.jackson-dataformat-cbor-2.9.8.jar

Thanks - Chris