Run ShardCoordinator in a Node with Different Role

Akka Akka Cluster
1

From reference.conf:

# Settings for the coordinator singleton. Same layout as akka.cluster.singleton.
# The "role" of the singleton configuration is not used. The singleton role will
# be the same as "akka.cluster.sharding.role".
# A lease can be configured in these settings for the coordinator singleton
coordinator-singleton = ${akka.cluster.singleton}

Which means that you cannot start a shard region in role1 with the coordinator-singleton being located somewhere in role2.

We use spot VMs heavily and if the oldest node is shutdown it creates a bigger disruption than with any other node. My idea was to introduce a non-spot VM to be responsible for shard coordinators and when looking up the config I realized that it is not possible. I could just add a non-spot VM and eventually it will always become the oldest node, but I don’t actually want to run any sharded actors on it, the goal would be to keep it as small as possible so that it only needs to coordinate the shards.

Is this an accidental limitation or would changing this behavior be very difficult?

2

I think such a use case/request never came up before, the use case I’ve seen roles used for in sharding has been splitting the cluster node responsibility between for example ingress/“frontend” nodes and “backend” nodes, or maybe workers for CPU bound actors that should not interfere with sharding.

For such scenarios separating out the coordinator does not make a lot of sense, so the behavior is intentional, however I don’t think it would be very complicated to allow separating it out with a role for the coordinator singleton (as long as the default behavior stays the same).

Feel free to try the idea out in a PR if you like.

3

Thanks, I will give it a go