Hello all,
We’ve upgraded our services to Lagom 1.4.11, and using the latest sbt-reactive-app 1.7.1.
One small issue we found today is that while RP is generating the kubernetes deployment YAML files assuming that play is listening on port 10000, the latest reactive cli is using the port 9000 as a default. However, this is not a major issue as you can quite easily override the http port in your application.conf, which we did.
After upgrading to the 1.4.11, none of our deployments were able to form the cluster sharding and I suspect this error is the issue:
Association with remote system [akka.tcp://application@10-8-0-186.coolapp-testing.pod.cluster.local:2552] has failed, address is now gated for [5000] ms. Reason: [Association failed with [akka.tcp://application@10-8-0-186.coolapp-testing.pod.cluster.local:2552]] Caused by: [The remote system explicitly disassociated (reason unknown).]
We still couldn’t really figure out what is causing this error. And we are still investigating this issue at the moment but sharing here first anyways, just in case someone has the clue for it.
Thanks,
The full log message is:
2019-02-13T04:25:00.675Z [e[37minfoe[0m] play.core.server.AkkaHttpServer [] - Listening for HTTP on /127.0.0.1:10000
2019-02-13T04:25:09.176Z [e[33mwarne[0m] akka.cluster.Cluster(akka://application) [sourceThread=application-akka.actor.default-dispatcher-5, akkaTimestamp=04:25:09.176UTC, akkaSource=akka.cluster.Cluster(akka://application), sourceActorSystem=application] - Cluster Node [akka.tcp://application@10-8-1-7.coolapp-testing.pod.cluster.local:2552] - Couldn't join seed nodes after [2] attempts, will try again. seed-nodes=[akka.tcp://application@10-8-0-186.coolapp-testing.pod.cluster.local:2552]
2019-02-13T04:25:09.347Z [e[33mwarne[0m] akka.remote.ReliableDeliverySupervisor [sourceThread=application-akka.remote.default-remote-dispatcher-23, akkaTimestamp=04:25:09.346UTC, akkaSource=akka.tcp://application@10-8-1-7.coolapp-testing.pod.cluster.local:2552/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2Fapplication%4010-8-0-186.coolapp-testing.pod.cluster.local%3A2552-0, sourceActorSystem=application] - Association with remote system [akka.tcp://application@10-8-0-186.coolapp-testing.pod.cluster.local:2552] has failed, address is now gated for [5000] ms. Reason: [Association failed with [akka.tcp://application@10-8-0-186.coolapp-testing.pod.cluster.local:2552]] Caused by: [The remote system explicitly disassociated (reason unknown).]
2019-02-13T04:25:14.225Z [e[33mwarne[0m] akka.cluster.Cluster(akka://application) [sourceThread=application-akka.actor.default-dispatcher-5, akkaTimestamp=04:25:14.224UTC, akkaSource=akka.cluster.Cluster(akka://application), sourceActorSystem=application] - Cluster Node [akka.tcp://application@10-8-1-7.coolapp-testing.pod.cluster.local:2552] - Couldn't join seed nodes after [3] attempts, will try again. seed-nodes=[akka.tcp://application@10-8-0-186.coolapp-testing.pod.cluster.local:2552]
2019-02-13T04:25:19.275Z [e[33mwarne[0m] akka.cluster.Cluster(akka://application) [sourceThread=application-akka.actor.default-dispatcher-15, akkaTimestamp=04:25:19.275UTC, akkaSource=akka.cluster.Cluster(akka://application), sourceActorSystem=application] - Cluster Node [akka.tcp://application@10-8-1-7.coolapp-testing.pod.cluster.local:2552] - Couldn't join seed nodes after [4] attempts, will try again. seed-nodes=[akka.tcp://application@10-8-0-186.coolapp-testing.pod.cluster.local:2552]
2019-02-13T04:25:19.432Z [e[33mwarne[0m] akka.remote.ReliableDeliverySupervisor [sourceThread=application-akka.remote.default-remote-dispatcher-24, akkaTimestamp=04:25:19.431UTC, akkaSource=akka.tcp://application@10-8-1-7.coolapp-testing.pod.cluster.local:2552/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2Fapplication%4010-8-0-186.coolapp-testing.pod.cluster.local%3A2552-0, sourceActorSystem=application] - Association with remote system [akka.tcp://application@10-8-0-186.coolapp-testing.pod.cluster.local:2552] has failed, address is now gated for [5000] ms. Reason: [Association failed with [akka.tcp://application@10-8-0-186.coolapp-testing.pod.cluster.local:2552]] Caused by: [The remote system explicitly disassociated (reason unknown).]
2019-02-13T04:25:24.305Z [e[33mwarne[0m] akka.cluster.Cluster(akka://application) [sourceThread=application-akka.actor.default-dispatcher-2, akkaTimestamp=04:25:24.304UTC, akkaSource=akka.cluster.Cluster(akka://application), sourceActorSystem=application] - Cluster Node [akka.tcp://application@10-8-1-7.coolapp-testing.pod.cluster.local:2552] - Couldn't join seed nodes after [5] attempts, will try again. seed-nodes=[akka.tcp://application@10-8-0-186.coolapp-testing.pod.cluster.local:2552]
2019-02-13T04:25:29.335Z [e[33mwarne[0m] akka.cluster.Cluster(akka://application) [sourceThread=application-akka.actor.default-dispatcher-21, akkaTimestamp=04:25:29.334UTC, akkaSource=akka.cluster.Cluster(akka://application), sourceActorSystem=application] - Cluster Node [akka.tcp://application@10-8-1-7.coolapp-testing.pod.cluster.local:2552] - Couldn't join seed nodes after [6] attempts, will try again. seed-nodes=[akka.tcp://application@10-8-0-186.coolapp-testing.pod.cluster.local:2552]
2019-02-13T04:25:29.514Z [e[33mwarne[0m] akka.remote.ReliableDeliverySupervisor [sourceThread=application-akka.remote.default-remote-dispatcher-23, akkaTimestamp=04:25:29.513UTC, akkaSource=akka.tcp://application@10-8-1-7.coolapp-testing.pod.cluster.local:2552/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2Fapplication%4010-8-0-186.coolapp-testing.pod.cluster.local%3A2552-0, sourceActorSystem=application] - Association with remote system [akka.tcp://application@10-8-0-186.coolapp-testing.pod.cluster.local:2552] has failed, address is now gated for [5000] ms. Reason: [Association failed with [akka.tcp://application@10-8-0-186.coolapp-testing.pod.cluster.local:2552]] Caused by: [The remote system explicitly disassociated (reason unknown).]
2019-02-13T04:25:34.354Z [e[33mwarne[0m] akka.cluster.Cluster(akka://application) [sourceThread=application-akka.actor.default-dispatcher-2, akkaTimestamp=04:25:34.354UTC, akkaSource=akka.cluster.Cluster(akka://application), sourceActorSystem=application] - Cluster Node [akka.tcp://application@10-8-1-7.coolapp-testing.pod.cluster.local:2552] - Couldn't join seed nodes after [7] attempts, will try again. seed-nodes=[akka.tcp://application@10-8-0-186.coolapp-testing.pod.cluster.local:2552]
2019-02-13T04:25:39.255Z [e[33mwarne[0m] akka.cluster.Cluster(akka://application) [sourceThread=application-akka.actor.default-dispatcher-15, akkaTimestamp=04:25:39.254UTC, akkaSource=akka.cluster.Cluster(akka://application), sourceActorSystem=application] - Cluster Node [akka.tcp://application@10-8-1-7.coolapp-testing.pod.cluster.local:2552] - Joining of seed-nodes [akka.tcp://application@10-8-0-186.coolapp-testing.pod.cluster.local:2552] was unsuccessful after configured shutdown-after-unsuccessful-join-seed-nodes [40000 milliseconds]. Running CoordinatedShutdown.
2019-02-13T04:25:39.281Z [e[37minfoe[0m] akka.cluster.singleton.ClusterSingletonManager [sourceThread=application-akka.actor.default-dispatcher-20, akkaTimestamp=04:25:39.281UTC, akkaSource=akka.tcp://application@10-8-1-7.coolapp-testing.pod.cluster.local:2552/system/sharding/AdminEntityCoordinator, sourceActorSystem=application] - Exited [akka.tcp://application@10-8-1-7.coolapp-testing.pod.cluster.local:2552]
2019-02-13T04:25:39.284Z [e[37minfoe[0m] akka.cluster.singleton.ClusterSingletonManager [sourceThread=application-akka.actor.default-dispatcher-3, akkaTimestamp=04:25:39.282UTC, akkaSource=akka.tcp://application@10-8-1-7.coolapp-testing.pod.cluster.local:2552/user/cassandraOffsetStorePrepare-singleton, sourceActorSystem=application] - Exited [akka.tcp://application@10-8-1-7.coolapp-testing.pod.cluster.local:2552]
2019-02-13T04:25:39.284Z [e[37minfoe[0m] akka.cluster.Cluster(akka://application) [sourceThread=application-akka.actor.default-dispatcher-20, akkaTimestamp=04:25:39.283UTC, akkaSource=akka.cluster.Cluster(akka://application), sourceActorSystem=application] - Cluster Node [akka.tcp://application@10-8-1-7.coolapp-testing.pod.cluster.local:2552] - Exiting completed
2019-02-13T04:25:39.289Z [e[37minfoe[0m] akka.cluster.Cluster(akka://application) [sourceThread=application-akka.actor.default-dispatcher-20, akkaTimestamp=04:25:39.287UTC, akkaSource=akka.cluster.Cluster(akka://application), sourceActorSystem=application] - Cluster Node [akka.tcp://application@10-8-1-7.coolapp-testing.pod.cluster.local:2552] - Shutting down...
2019-02-13T04:25:39.292Z [e[37minfoe[0m] akka.cluster.Cluster(akka://application) [sourceThread=application-akka.actor.default-dispatcher-20, akkaTimestamp=04:25:39.289UTC, akkaSource=akka.cluster.Cluster(akka://application), sourceActorSystem=application] - Cluster Node [akka.tcp://application@10-8-1-7.coolapp-testing.pod.cluster.local:2552] - Successfully shut down