Akka Sharding - Event Sourcing - Performance Issue

Hi Team,

I am using the latest Akka 2.6.4 to implement Akka Cluster Sharding with Event Sourcing.
I had made a small application while implementing Akka Http to push the request to an application to create a persistent entity actor upon receiving requests.
Inside those entities, I am preserving some data while fetching from DB to in memory to read on later scenarios to improve the performance of my application.

I had created a cluster with 2 nodes so that whenever I am pushing the request, the entities are getting created in either of the nodes, and periodically rebalancing also takes place of all the persistent entities.

The issue what I am facing is whenever I am firing HTTP GET request to the node and if the entity exists on the same node where I am firing the request, it is taking around 2 ms to send the response back(in case if I am firing around 50 concurrent requests), which is quite promising but on the other hand, if I am firing HTTP GET request to the node and entity exist on another node of the cluster then it is taking around 6ms to send back the response which is almost hike of 200% with respect to response time.

Attaching the apache benchmark output for both types of entities

Entity that exists on same node

__Concurrency Level: 50
Time taken for tests: 3.943 seconds
Complete requests: 100000
Time per request: 1.971 [ms] (mean)
Time per request: 0.039 [ms] (mean, across all concurrent requests)

Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 0 0.2 0 10
Processing: 0 2 1.4 2 37
Waiting: 0 2 1.4 2 37
Total: 0 2 1.5 2 37

Percentage of the requests served within a certain time (ms)
50% 2
66% 2
75% 2
80% 2
90% 3
95% 4
98% 5
99% 7
100% 37 (longest request)_

Entity that exists on different node

_Concurrency Level: 50
Time taken for tests: 10.848 seconds
Time per request: 5.424 [ms] (mean)
Time per request: 0.108 [ms] (mean, across all concurrent requests)

Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 0 0.2 0 17
Processing: 1 5 2.2 5 56
Waiting: 1 5 2.2 5 39
Total: 1 5 2.2 5 57

Percentage of the requests served within a certain time (ms)
50% 5
66% 6
75% 6
80% 7
90% 8
95% 9
98% 10
99% 12
100% 57 (longest request)_

Kindly have a look and share your valuable feedback

Thanks