Sorry, my bad stating that, the
AdaptiveLoadBalancing doesn’t do any dynamic resizing, except for when you remove and add nodes to the cluster.
If automatic rebalance would be problematic with your stream, what do you do when a new node is added to your cluster, or you need to remove the node a stream is running on, or when a node have crashed?
Additionally choosing the right node™ to run a stream on is trickier than choosing the right node to run a small:ish workload on, since a stream is in general long lived, imagine five requests coming at roughly the same time, all seeing node N having low load and starting long lived streams on that node.
Best might be to make sure you have a way to shard requests fairly on some identifier/property of the request.
There’s a singleton + sharding cluster tool available in Lagom which is used to essentially keep singletons alive (and consume persistence queries sharded by tag) but spread out across nodes, maybe this could be something to look into. We have some plans to port this to a pure Akka cluster tool but for now you’d have to mimic it yourself. (Sources are here: https://github.com/lagom/lagom/blob/master/persistence/core/src/main/scala/com/lightbend/lagom/internal/persistence/cluster/ClusterDistribution.scala)
I think I mentioned in the gitter discussion a couple of days ago, but you will also have to think about back pressure and not overwhelming the worker nodes. Whatever solution you choose, make sure you try a higher inflow of work than you can complete out and see in what way that affects the solution. (This is usually why you should avoid routers for distributing workloads)