Actor slowness in Akka Cluster when multiple threads send messages to ShardRegion with large Message size

Akka Akka Cluster
, ,
1

We implemented Akka clustering with cluster sharding for a use case.
When we doing load testing for that, We created 1000 entity actors in a Node by cluster Sharding. And we sends messages to that entity actors from a Proxy Node (Shard Region Proxy on other Node).

Kindly check the update Section on the question too

What we done

Using
akka.cluster.shard.remember.entities=ddata

And dedicated dispatcher given for entity actors with 1000 threads too.
We produced from single thread and multiple threads to check the behavior.

The Issue we faced

When sending from a single thread in Proxy Node the delay time in Actor Receive method in Entity actor is very minimal.

But when sending messages to the entity actors with multiple threads (same no of messages) the delay time is high (>100ms) with respect to no of threads using.

Which is similar when we use ActorSelection in Proxy Node. When checking for an entity actor availability with ActorSelection in Proxy node. Based on no of parallel threads that delay too gets increasing.

Need some clarification on

How the internal mechanism works when sending messages to entity actors in ShardRegion proxy with multiple threads.

  • Is there any performance tuning has to be done or i missing any important configuration.?
  • Why the delay is increasing when more no of parallel threads tries to send messages to Cluster entity actors compared to single thread sending.?

Lets say an example…

I have 1000 entity actors spitted evenly to 3 shards.
And I’m sending messages to entity actors with 100 threads, each threads sends a message to 10 entity actors.
After sending a message successfully then only next message will be send.
So at a time there are only 100 messages will send to that ShardRegion in parallel.

I will get reply from an entity actor after successful reply by using Pattern.ask

After getting reply only next message send to another entity.

First messages are received by the entity actors instantly but when consecutive messages the delay between proxy to entity Actor on receive is getting increase.

Update on the Question

The actual reason for the delay is Message size.When i send 1 KB message with 100 threads there is not much delay.
But When i send 100KB message from 100 Threads the delay is high.
By reading some blogs and akka documentation and reference.conf find out some params

akka {
 remote {
  netty.tcp {
   # Sets the send buffer size of the Sockets,
   # set to 0b for platform default
   send-buffer-size = 256000b

   # Sets the receive buffer size of the Sockets,
   # set to 0b for platform default
   receive-buffer-size = 256000b
  }
 }
}

We have increased this from 2 MB to even 100MB but there is no gain in the performance in large message size.

Is that i’m missing any important configuration ?

2

So you don’t have enough information to really answer this. You probably need to post a reproduction if you want help as this is going to be very specific to your environment, perf test suites, and expectations. Frankly, my initial guess is that you have a poorly written test suite.

1 Like
3

Thank you for making a response. I understood my question is not good enough. https://groups.google.com/g/akka-user/c/szVVa7Ue5XI here is the issue which i faced. But even after increasing send and receive buffer size i’m not gaining any performance.

4

Those send and receive buffer sizes are, as far as I know, only used in classic remoting, not Artery. So if you’re using Artery (which has been the default for a number of years), those settings should have no effect.

1 Like
5

Im not using Artery im using classic remoting only. The transport im using is netty.tcp.

6

I configured hostname and port as inside netty.tcp. I’m using akka 2.5.32 version