Akka Cluster Shards are getting placed in a single node

I’m using a 50 node cluster to handle stream data and creating child actors using service actors which are handling streams. Whenever I start a single stream, all the child actors shards are placed on the same node as the service actor handling stream.

The same program run easily on my laptop .

This is the log of the node handling the stream.
[

INFO] [03/29/2018 19:24:56.650] [Segmenter-akka.actor.default-dispatcher-37] [akka.tcp://Segmenter@10.244.125.8:2551/system/sharding/customerActor] ShardRegion for [customerActor] is using [10.0 %] of its buffer capacity.
[INFO] [03/29/2018 19:25:00.038] [Segmenter-akka.actor.default-dispatcher-2] [akka.tcp://Segmenter@10.244.125.8:2551/system/sharding/customerActor] ShardRegion for [customerActor] is using [20.0 %] of its buffer capacity.
[INFO] [03/29/2018 19:25:01.908] [Segmenter-akka.actor.default-dispatcher-36] [akka.tcp://Segmenter@10.244.125.8:2551/system/sharding/customerActor] ShardRegion for [customerActor] is using [30.0 %] of its buffer capacity.
[INFO] [03/29/2018 19:25:03.283] [Segmenter-akka.actor.default-dispatcher-5] [akka.tcp://Segmenter@10.244.125.8:2551/system/sharding/customerActor] ShardRegion for [customerActor] is using [40.0 %] of its buffer capacity.
[INFO] [03/29/2018 19:25:07.808] [Segmenter-akka.actor.default-dispatcher-2] [akka.tcp://Segmenter@10.244.125.8:2551/system/sharding/customerActor] ShardRegion for [customerActor] is using [50.0 %] of its buffer capacity.
[WARN] [03/29/2018 19:25:08.815] [Segmenter-akka.actor.default-dispatcher-3] [akka.tcp://Segmenter@10.244.125.8:2551/system/sharding/customerActor] ShardRegion for [customerActor] is using [60.0 %] of its buffer capacity. The coordinator might not be available. You might want to check cluster membership status.
[WARN] [03/29/2018 19:25:10.306] [Segmenter-akka.actor.default-dispatcher-3] [akka.tcp://Segmenter@10.244.125.8:2551/system/sharding/customerActor] ShardRegion for [customerActor] is using [70.0 %] of its buffer capacity. The coordinator might not be available. You might want to check cluster membership status.
[WARN] [03/29/2018 19:25:11.265] [Segmenter-akka.actor.default-dispatcher-6] [akka.tcp://Segmenter@10.244.125.8:2551/system/sharding/customerActor] ShardRegion for [customerActor] is using [80.0 %] of its buffer capacity. The coordinator might not be available. You might want to check cluster membership status.
[WARN] [03/29/2018 19:25:14.693] [Segmenter-akka.actor.default-dispatcher-2] [akka.tcp://Segmenter@10.244.125.8:2551/system/sharding/customerActor] ShardRegion for [customerActor] is using [80.0 %] of its buffer capacity. The coordinator might not be available. You might want to check cluster membership status.
[WARN] [03/29/2018 19:25:15.781] [Segmenter-akka.actor.default-dispatcher-6] [akka.tcp://Segmenter@10.244.125.8:2551/system/sharding/customerActor] ShardRegion for [customerActor] is using [90.0 %] of its buffer capacity. The coordinator might not be available. You might want to check cluster membership status.
[WARN] [03/29/2018 19:25:17.348] [Segmenter-akka.actor.default-dispatcher-37] [akka.tcp://Segmenter@10.244.125.8:2551/system/sharding/customerActor] ShardRegion for [customerActor] is using [100.0 %] of its buffer capacity. The coordinator might not be available. You might want to check cluster membership status.
[WARN] [03/29/2018 19:25:17.348] [Segmenter-akka.actor.default-dispatcher-37] [akka.tcp://Segmenter@10.244.125.8:2551/system/sharding/customerActor] Buffer is full, dropping message for shard [161]
[INFO] [03/29/2018 19:25:17.349] [Segmenter-akka.actor.default-dispatcher-6] [akka://Segmenter/deadLetters] Message [starbucks.bits.segmenter.models.Customer] from Actor[akka://Segmenter/system/sharding/customerActor#-283448532] to Actor[akka://Segmenter/deadLetters] was not delivered. [2] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.
[INFO] [03/29/2018 19:25:17.356] [Segmenter-akka.actor.default-dispatcher-2] [akka://Segmenter/deadLetters] Message [starbucks.bits.segmenter.models.Customer] from Actor[akka://Segmenter/system/sharding/customerActor#-283448532] to Actor[akka://Segmenter/deadLetters] was not delivered. [3] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.
[INFO] [03/29/2018 19:25:17.365] [Segmenter-akka.actor.default-dispatcher-38] [akka://Segmenter/deadLetters] Message [starbucks.bits.segmenter.models.Customer] from Actor[akka://Segmenter/system/sharding/customerActor#-283448532] to Actor[akka://Segmenter/deadLetters] was not delivered. [4] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.
[INFO] [03/29/2018 19:25:17.366] [Segmenter-akka.actor.default-dispatcher-36] [akka://Segmenter/deadLetters] Message [starbucks.bits.segmenter.models.Customer] from Actor[akka://Segmenter/system/sharding/customerActor#-283448532] to Actor[akka://Segmenter/deadLetters] was not delivered. [5] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.
[INFO] [03/29/2018 19:25:17.366] [Segmenter-akka.actor.default-dispatcher-36] [akka://Segmenter/deadLetters] Message [starbucks.bits.segmenter.models.Customer] from Actor[akka://Segmenter/system/sharding/customerActor#-283448532] to Actor[akka://Segmenter/deadLetters] was not delivered. [6] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.
[INFO] [03/29/2018 19:25:17.372] [Segmenter-akka.actor.default-dispatcher-36] [akka://Segmenter/deadLetters] Message [starbucks.bits.segmenter.models.Customer] from Actor[akka://Segmenter/system/sharding/customerActor#-283448532] to Actor[akka://Segmenter/deadLetters] was not delivered. [7] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.
[INFO] [03/29/2018 19:25:17.372] [Segmenter-akka.actor.default-dispatcher-36] [akka://Segmenter/deadLetters] Message [starbucks.bits.segmenter.models.Customer] from Actor[akka://Segmenter/system/sharding/customerActor#-283448532] to Actor[akka://Segmenter/deadLetters] was not delivered. [8] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.
[INFO] [03/29/2018 19:25:17.372] [Segmenter-akka.actor.default-dispatcher-36] [akka://Segmenter/deadLetters] Message [starbucks.bits.segmenter.models.Customer] from Actor[akka://Segmenter/system/sharding/customerActor#-283448532] to Actor[akka://Segmenter/deadLetters] was not delivered. [9] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.
[INFO] [03/29/2018 19:25:17.372] [Segmenter-akka.actor.default-dispatcher-5] [akka://Segmenter/deadLetters] Message [starbucks.bits.segmenter.models.Customer] from Actor[akka://Segmenter/system/sharding/customerActor#-283448532] to Actor[akka://Segmenter/deadLetters] was not delivered. [10] dead letters encountered, no more dead letters will be logged. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.
[WARN] [03/29/2018 19:25:18.422] [Segmenter-akka.actor.default-dispatcher-36] [akka.tcp://Segmenter@10.244.125.8:2551/system/sharding/customerActor] ShardRegion for [customerActor] is using [100.0 %] of its buffer capacity. The coordinator might not be available. You might want to check cluster membership status.
[WARN] [03/29/2018 19:25:18.422] [Segmenter-akka.actor.default-dispatcher-36] [akka.tcp://Segmenter@10.244.125.8:2551/system/sharding/customerActor] Buffer is full, dropping message for shard [325]
[WARN] [03/29/2018 19:25:19.269] [Segmenter-akka.actor.default-dispatcher-38] [akka.tcp://Segmenter@10.244.125.8:2551/system/sharding/customerActor] ShardRegion for [customerActor] is using [100.0 %] of its buffer capacity. The coordinator might not be available. You might want to check cluster membership status.
[WARN] [03/29/2018 19:25:19.274] [Segmenter-akka.actor.default-dispatcher-38] [akka.tcp://Segmenter@10.244.125.8:2551/system/sharding/customerActor] Buffer is full, dropping message for shard [163]
[WARN] [03/29/2018 19:25:20.463] [Segmenter-akka.actor.default-dispatcher-6] [akka.tcp://Segmenter@10.244.125.8:2551/system/sharding/customerActor] ShardRegion for [customerActor] is using [100.0 %] of its buffer capacity. The coordinator might not be available. You might want to check cluster membership status.
[WARN] [03/29/2018 19:25:20.463] [Segmenter-akka.actor.default-dispatcher-6] [akka.tcp://Segmenter@10.244.125.8:2551/system/sharding/customerActor] Buffer is full, dropping message for shard [319]
[WARN] [03/29/2018 19:25:21.242] [Segmenter-akka.actor.default-dispatcher-2] [akka.tcp://Segmenter@10.244.125.8:2551/system/sharding/customerActor] Buffer is full, dropping message for shard [99]
[WARN] [03/29/2018 19:25:21.355] [Segmenter-akka.actor.default-dispatcher-5] [akka.tcp://Segmenter@10.244.125.8:2551/system/sharding/customerActor] ShardRegion for [customerActor] is using [100.0 %] of its buffer capacity. The coordinator might not be available. You might want to check cluster membership status.
[WARN] [03/29/2018 19:25:21.355] [Segmenter-akka.actor.default-dispatcher-5] [akka.tcp://Segmenter@10.244.125.8:2551/system/sharding/customerActor] Buffer is full, dropping message for shard [301]
[WARN] [03/29/2018 19:25:21.358] [Segmenter-akka.actor.default-dispatcher-38] [akka.tcp://Segmenter@10.244.125.8:2551/system/sharding/customerActor] Buffer is full, dropping message for shard [100]
[WARN] [03/29/2018 19:25:21.360] [Segmenter-akka.actor.default-dispatcher-5] [akka.tcp://Segmenter@10.244.125.8:2551/system/sharding/customerActor] Buffer is full, dropping message for shard [107]
[WARN] [03/29/2018 19:25:22.619] [Segmenter-akka.actor.default-dispatcher-5] [akka.tcp://Segmenter@10.244.125.8:2551/system/sharding/customerActor] ShardRegion for [customerActor] is using [100.0 %] of its buffer capacity. The coordinator might not be available. You might want to check cluster membership status.
[WARN] [03/29/2018 19:25:22.619] [Segmenter-akka.actor.default-dispatcher-5] [akka.tcp://Segmenter@10.244.125.8:2551/system/sharding/customerActor] Buffer is full, dropping message for shard [3]
[WARN] [03/29/2018 19:25:23.453] [Segmenter-akka.actor.default-dispatcher-3] [akka.tcp://Segmenter@10.244.125.8:2551/system/sharding/customerActor] ShardRegion for [customerActor] is using [100.0 %] of its buffer capacity. The coordinator might not be available. You might want to check cluster membership status.
[WARN] [03/29/2018 19:25:23.453] [Segmenter-akka.actor.default-dispatcher-3] [akka.tcp://Segmenter@10.244.125.8:2551/system/sharding/customerActor] Buffer is full, dropping message for shard [338]
[WARN] [03/29/2018 19:25:27.131] [Segmenter-akka.actor.default-dispatcher-36] [akka.tcp://Segmenter@10.244.125.8:2551/system/sharding/customerActor] ShardRegion for [customerActor] is using [100.0 %] of its buffer capacity. The coordinator might not be available. You might want to check cluster membership status.
[WARN] [03/29/2018 19:25:27.131] [Segmenter-akka.actor.default-dispatcher-36] [akka.tcp://Segmenter@10.244.125.8:2551/system/sharding/customerActor] Buffer is full, dropping message for shard [30]
[WARN] [03/29/2018 19:25:27.724] [Segmenter-akka.actor.default-dispatcher-37] [akka.tcp://Segmenter@10.244.125.8:2551/system/sharding/customerActor] ShardRegion for [customerActor] is using [100.0 %] of its buffer capacity. The coordinator might not be available. You might want to check cluster membership status.
[WARN] [03/29/2018 19:25:27.724] [Segmenter-akka.actor.default-dispatcher-37] [akka.tcp://Segmenter@10.244.125.8:2551/system/sharding/customerActor] Buffer is full, dropping message for shard [290]
[WARN] [03/29/2018 19:25:27.795] [Segmenter-akka.actor.default-dispatcher-4] [akka.tcp://Segmenter@10.244.125.8:2551/system/sharding/customerActor] Buffer is full, dropping message for shard [254]
[WARN] [03/29/2018 19:25:27.809] [Segmenter-akka.actor.default-dispatcher-36] [akka.tcp://Segmenter@10.244.125.8:2551/system/sharding/customerActor] Buffer is full, dropping message for shard [56]
[WARN] [03/29/2018 19:25:27.828] [Segmenter-akka.actor.default-dispatcher-2] [akka.tcp://Segmenter@10.244.125.8:2551/system/sharding/customerActor] Buffer is full, dropping message for shard [257]
[WARN] [03/29/2018 19:25:27.833] [Segmenter-akka.actor.default-dispatcher-4] [akka.tcp://Segmenter@10.244.125.8:2551/system/sharding/customerActor] Buffer is full, dropping message for shard [163]
[WARN] [03/29/2018 19:25:27.880] [Segmenter-akka.actor.default-dispatcher-4] [akka.tcp://Segmenter@10.244.125.8:2551/system/sharding/customerActor] Buffer is full, dropping message for shard [112]
[WARN] [03/29/2018 19:25:27.893] [Segmenter-akka.actor.default-dispatcher-36] [akka.tcp://Segmenter@10.244.125.8:2551/system/sharding/customerActor] Buffer is full, dropping message for shard [326]
[WARN] [03/29/2018 19:25:27.894] [Segmenter-akka.actor.default-dispatcher-5] [akka.tcp://Segmenter@10.244.125.8:2551/system/sharding/customerActor] Buffer is full, dropping message for shard [146]
[WARN] [03/29/2018 19:25:27.898] [Segmenter-akka.actor.default-dispatcher-3] [akka.tcp://Segmenter@10.244.125.8:2551/system/sharding/customerActor] Buffer is full, dropping message for shard [194]

Share some code? Hard to see what’s wrong otherwise.

Are you starting cluster starting (the extension) on all nodes? That’s required

The logs with prefix ”Cluster Node” would be interesting, e.g. to verify that the nodes join the cluster and are moved to the Up status.