We’ve switched to kubernetes-api for clusrter bootstraping, and so far it looks great except one thing - upon starting the cluster up (we do blue/green deployments) services that start first spit out a lot of messages like these:
2018-10-04 09:10:28.004 [ERROR] akka.management.cluster.bootstrap.internal.HttpContactPointBootstrap - Probing [http://10-48-3-215.live.pod.cluster.local:19998/bootstrap/seed-nodes] failed due to: Tcp command [Connect(10-48-3-215.live.pod.cluster.local:19998,None,List(),Some(10 seconds),true)] failed because of java.net.ConnectException: Connection refused
My understanding of why this happens:
- first of the nodes starts up and checks kubernetes labels - it sees other nodes which are still starting up
- those other nodes are still in the startup phase, and management port is not yet ready (we do ClusterBootstrap.start() and AkkaManagement.start() at the very last part of sequence init so that everything would be as much ready as possible before joining the cluster)
- we see a lot of errors.
To me this HttpContactPointBootstrap probing failure looks like a normal flow - service is not ready, will try again later, though it is logged as error as if you would not something to act upon