What is the recommended deployment/upgrade strategy for clustered Lagom services in kubernetes?
Currently, every time we deploy to k8s, we start up new nodes and allow them to form a new cluster. Once the new cluster is formed, the k8s readiness checks will pass, therefore k8s will consider those nodes as ready to receive traffic and will start to terminate old nodes. Can anyone see an issue with this strategy? Are we exposing ourselves to split-brain risk?
K8s seems to instantly kill old cluster when the new cluster is formed but I do worry it might not be always be the case.
The alternative strategy would be to add new nodes to the existing cluster in a rolling upgrade till all nodes have been replaced. The reason we didn’t go with this strategy was because of concerns regarding ser/deser issues that might happen when new nodes communicate with old ones.
Thoughts and suggestions are very appreciated.