We implemented Akka clustering with cluster sharding for a use case.
When we doing load testing for that, We created 1000 entity actors in a Node by cluster Sharding. And we sends messages to that entity actors from a Proxy Node (Shard Region Proxy on other Node).
Using
akka.cluster.shard.remember.entities=ddata
And dedicated dispatcher given for entity actors with 1000 threads too.
When sending from a single thread in Proxy Node the delay time in Actor Receive method in Entity actor is very minimal.
But when sending messages to the entity actors with multiple threads (same no of messages) the delay time is high (>100ms) with respect to no of threads using.
Which is similar when we use ActorSelection in Proxy Node. When checking for an entity actor availability with ActorSelection in Proxy node. Based on no of parallel threads that delay too gets increasing.
How the internal mechanism works when sending messages to entity actors in ShardRegion proxy with multiple threads.
- Is there any performance tuning has to be done or i missing any important configuration.?
- Why the delay is increasing when more no of parallel threads tries to send messages to Cluster entity actors compared to single thread sending.?
Lets say an example…
I have 1000 entity actors spitted evenly to 3 shards.
And I’m sending messages to entity actors with 100 threads, each threads sends a message to 10 entity actors.
After sending a message successfully then only next message will be send.
So at a time there are only 100 messages will send to that ShardRegion in parallel.
I will get reply from an entity actor after successful reply by using Pattern.ask
After getting reply only next message send to another entity.
First messages are received by the entity actors instantly but when consecutive messages the delay between proxy to entity Actor on receive is getting increase.