So, the nodes(11.128.83.43, 11.128.83.33, 11.128.83.32) make up a cluster， 11.128.83.43 is member-1, 11.128.83.33 is member-2 and 11.128.83.32 is member-3. these days I want to replace the origin nodes with three new nodes(11.128.83.41, 11.128.83.48, 11.128.83.49 ) ,
My approach is to expand the cluster into six nodes(11.128.83.43, 11.128.83.33, 11.128.83.32, 11.128.83.41, 11.128.83.48, 11.128.83.49 ) first, and the next step I will shutdown the old nodes(11.128.83.43, 11.128.83.33, 11.128.83.32), the remaining nodes will be the new. so i’m wandering if the akka will allow me to update and reload the config without shutdown the cluster service?
I searched some akka docs, there is no related anwser
Yes, you can add more addresses to the seed-nodes configuration. The new nodes will join the existing cluster. Then you can stop the old nodes and remove them from the seed-nodes.
It will try to connect to all seed-nodes but it’s enough that one of them answers to join the existing cluster.
Btw, akka.tcp indicates that this is a very old version of Akka, but maybe that is a requirement from ODL? Otherwise you should plan for migrating to later version, see Migration Guide 2.5.x to 2.6.x • Akka Documentation
As you mentioned, The first step I shoud replace the three seed nodes with the new six nodes, I’m wandering if this transformation need to stop the service?
And I will try the following steps to verify the feasibility:
Check the old cluster is running well with old seed nodes(11.128.83.43, 11.128.83.33, 11.128.83.32)
Update akka Configuration and restart old nodes:
2.1. add three nodes (11.128.83.43, 11.128.83.33, 11.128.83.32, 11.128.83.41, 11.128.83.48, 11.128.83.49 ) into the akka config of old seed nodes(11.128.83.43, 11.128.83.33, 11.128.83.32).
2.2. restart the old three nodes recursively I think this step will not stop the system ?
start the three new seed nodes. I will start 11.128.83.41, 11.128.83.48, 11.128.83.49 as new seed nodes which have the same akka config with old three nodes.
stop the three old seed nodes(11.128.83.43, 11.128.83.33, 11.128.83.32)
Update akka Configuration and restart new nodes:
5.1. delete three nodes (11.128.83.41, 11.128.83.48, 11.128.83.49 ) from the akka config
5.2. restart the new three nodes recursively(11.128.83.41, 11.128.83.48, 11.128.83.49)
About akka.tcp.
thanks for your remind, we are considering the upgradation of the akka system. and firstly we will replace the old nodes.
@leviramsey
you mean I can add three new nodes with the akka config (seed nodes : 11.128.83.41, 11.128.83.48, 11.128.83.49) , and old node config no need to be update ?
But once a node has joined the cluster, the seed node configuration on that node doesn’t matter (and there’s no need for any nodes that have joined the cluster to have the same seed nodes).
So the first step, for zero downtime, is to join the new nodes to the existing cluster, using the current seed nodes. Once those have joined, take down the old nodes. At this point, you have moved the cluster to the three new nodes.
After this, if you wanted to join more nodes to the cluster, those nodes would need to have at least one node in the cluster at that time configured as a seed node. Adding nodes never requires changing configuration on the existing nodes in the cluster (though as long as the node’s configuration has a seed node that’s in the cluster, if its restarted, it will rejoin).
@leviramsey. thanks for your reply, i still have some questions:
So the first step, for zero downtime, is to join the new nodes to the existing cluster, using the current seed nodes. Once those have joined, take down the old nodes. At this point, you have moved the cluster to the three new nodes.
thanks, I get it, I’m trying it out
After this, if you wanted to join more nodes to the cluster, those nodes would need to have at least one node in the cluster at that time configured as a seed node
after the above steps, I think the cluster consists of the three new nodes(11.128.83.41, 11.128.83.48, 11.128.83.49), but there seed nodes of akka config are old nodes(11.128.83.43, 11.128.83.33, 11.128.83.32). I have the following two questions:
In this state, this cluster is working with wrong seed nodes in the akka config. how could a node rejoin the cluster when it restarted accidentally this time? I think the nodes will try to connect the seed nodes which is wrong. So… how can this node find the new nodes?
If i want to replace the old seed nodes with the new right ones, I will update the akka config and restart nodes one by one. Is this method Ok?
