Thanks a lot for your reply. I agree that that the current implementation is fine, and practical. Unless the coordinator singleton can ever become a bottleneck, there is no need to change it.
Also, don’t let me waste your time if this is unproductive. But here is what I think:
handover to make sure the old and new actor are not alive
Both the old node and the new node would know when the handover of a shard is completed. Between those two nodes it could be ensured that only one instance of the shard (entities) is active. Now, if another node mistakenly sends a message to the old node (after handover completed) or to the new node (before handover completed), the receiving node could forward that message to the other, correct node, since it knows that the actor lives there. This principle could be universally applied to redirect falsely addressed messages, because each node knows exactly which shards it controls. If it receives a wrongly addressed message, it just has to send it to the node it believes to be correct.
During handover, both nodes would stack incoming messages, and then send them to the correct node once handover is complete.
The mapping function would still take the place of the coordinator, which would have the mentioned scalability benefits.
you’d also need some form of coordination tracking the topology
I thought that functionality is already implemented by Akka, no?
leader coordinate this? The leader’s job is already to track cluster events, so there would not be any issue in respect to order of events.