I was trying out an experiment with Persistent actors on a cluster, backed by Cassandra, with a million actors. I had 3 nodes with 9 shards, and the throughput was lower than what I was expecting. Until, I found this line in the documentation:
As a rule of thumb, the number of shards should be a factor ten greater than the planned maximum number of cluster nodes. It doesn’t have to be exact
After I changed the number of shards to 30, the throughput improved significantly (almost 5 times).
My question is, what is happening under the hood that caused this, and where does this 10x guideline come from?