I’m using a 50 node cluster to handle stream data and creating child actors using service actors which are handling streams. Whenever I start a single stream, all the child actors shards are placed on the same node as the service actor handling stream.
The same program run easily on my laptop .
This is the log of the node handling the stream.
[
INFO] [03/29/2018 19:24:56.650] [Segmenter-akka.actor.default-dispatcher-37] [akka.tcp://Segmenter@10.244.125.8:2551/system/sharding/customerActor] ShardRegion for [customerActor] is using [10.0 %] of its buffer capacity.
[INFO] [03/29/2018 19:25:00.038] [Segmenter-akka.actor.default-dispatcher-2] [akka.tcp://Segmenter@10.244.125.8:2551/system/sharding/customerActor] ShardRegion for [customerActor] is using [20.0 %] of its buffer capacity.
[INFO] [03/29/2018 19:25:01.908] [Segmenter-akka.actor.default-dispatcher-36] [akka.tcp://Segmenter@10.244.125.8:2551/system/sharding/customerActor] ShardRegion for [customerActor] is using [30.0 %] of its buffer capacity.
[INFO] [03/29/2018 19:25:03.283] [Segmenter-akka.actor.default-dispatcher-5] [akka.tcp://Segmenter@10.244.125.8:2551/system/sharding/customerActor] ShardRegion for [customerActor] is using [40.0 %] of its buffer capacity.
[INFO] [03/29/2018 19:25:07.808] [Segmenter-akka.actor.default-dispatcher-2] [akka.tcp://Segmenter@10.244.125.8:2551/system/sharding/customerActor] ShardRegion for [customerActor] is using [50.0 %] of its buffer capacity.
[WARN] [03/29/2018 19:25:08.815] [Segmenter-akka.actor.default-dispatcher-3] [akka.tcp://Segmenter@10.244.125.8:2551/system/sharding/customerActor] ShardRegion for [customerActor] is using [60.0 %] of its buffer capacity. The coordinator might not be available. You might want to check cluster membership status.
[WARN] [03/29/2018 19:25:10.306] [Segmenter-akka.actor.default-dispatcher-3] [akka.tcp://Segmenter@10.244.125.8:2551/system/sharding/customerActor] ShardRegion for [customerActor] is using [70.0 %] of its buffer capacity. The coordinator might not be available. You might want to check cluster membership status.
[WARN] [03/29/2018 19:25:11.265] [Segmenter-akka.actor.default-dispatcher-6] [akka.tcp://Segmenter@10.244.125.8:2551/system/sharding/customerActor] ShardRegion for [customerActor] is using [80.0 %] of its buffer capacity. The coordinator might not be available. You might want to check cluster membership status.
[WARN] [03/29/2018 19:25:14.693] [Segmenter-akka.actor.default-dispatcher-2] [akka.tcp://Segmenter@10.244.125.8:2551/system/sharding/customerActor] ShardRegion for [customerActor] is using [80.0 %] of its buffer capacity. The coordinator might not be available. You might want to check cluster membership status.
[WARN] [03/29/2018 19:25:15.781] [Segmenter-akka.actor.default-dispatcher-6] [akka.tcp://Segmenter@10.244.125.8:2551/system/sharding/customerActor] ShardRegion for [customerActor] is using [90.0 %] of its buffer capacity. The coordinator might not be available. You might want to check cluster membership status.
[WARN] [03/29/2018 19:25:17.348] [Segmenter-akka.actor.default-dispatcher-37] [akka.tcp://Segmenter@10.244.125.8:2551/system/sharding/customerActor] ShardRegion for [customerActor] is using [100.0 %] of its buffer capacity. The coordinator might not be available. You might want to check cluster membership status.
[WARN] [03/29/2018 19:25:17.348] [Segmenter-akka.actor.default-dispatcher-37] [akka.tcp://Segmenter@10.244.125.8:2551/system/sharding/customerActor] Buffer is full, dropping message for shard [161]
[INFO] [03/29/2018 19:25:17.349] [Segmenter-akka.actor.default-dispatcher-6] [akka://Segmenter/deadLetters] Message [starbucks.bits.segmenter.models.Customer] from Actor[akka://Segmenter/system/sharding/customerActor#-283448532] to Actor[akka://Segmenter/deadLetters] was not delivered. [2] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.
[INFO] [03/29/2018 19:25:17.356] [Segmenter-akka.actor.default-dispatcher-2] [akka://Segmenter/deadLetters] Message [starbucks.bits.segmenter.models.Customer] from Actor[akka://Segmenter/system/sharding/customerActor#-283448532] to Actor[akka://Segmenter/deadLetters] was not delivered. [3] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.
[INFO] [03/29/2018 19:25:17.365] [Segmenter-akka.actor.default-dispatcher-38] [akka://Segmenter/deadLetters] Message [starbucks.bits.segmenter.models.Customer] from Actor[akka://Segmenter/system/sharding/customerActor#-283448532] to Actor[akka://Segmenter/deadLetters] was not delivered. [4] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.
[INFO] [03/29/2018 19:25:17.366] [Segmenter-akka.actor.default-dispatcher-36] [akka://Segmenter/deadLetters] Message [starbucks.bits.segmenter.models.Customer] from Actor[akka://Segmenter/system/sharding/customerActor#-283448532] to Actor[akka://Segmenter/deadLetters] was not delivered. [5] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.
[INFO] [03/29/2018 19:25:17.366] [Segmenter-akka.actor.default-dispatcher-36] [akka://Segmenter/deadLetters] Message [starbucks.bits.segmenter.models.Customer] from Actor[akka://Segmenter/system/sharding/customerActor#-283448532] to Actor[akka://Segmenter/deadLetters] was not delivered. [6] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.
[INFO] [03/29/2018 19:25:17.372] [Segmenter-akka.actor.default-dispatcher-36] [akka://Segmenter/deadLetters] Message [starbucks.bits.segmenter.models.Customer] from Actor[akka://Segmenter/system/sharding/customerActor#-283448532] to Actor[akka://Segmenter/deadLetters] was not delivered. [7] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.
[INFO] [03/29/2018 19:25:17.372] [Segmenter-akka.actor.default-dispatcher-36] [akka://Segmenter/deadLetters] Message [starbucks.bits.segmenter.models.Customer] from Actor[akka://Segmenter/system/sharding/customerActor#-283448532] to Actor[akka://Segmenter/deadLetters] was not delivered. [8] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.
[INFO] [03/29/2018 19:25:17.372] [Segmenter-akka.actor.default-dispatcher-36] [akka://Segmenter/deadLetters] Message [starbucks.bits.segmenter.models.Customer] from Actor[akka://Segmenter/system/sharding/customerActor#-283448532] to Actor[akka://Segmenter/deadLetters] was not delivered. [9] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.
[INFO] [03/29/2018 19:25:17.372] [Segmenter-akka.actor.default-dispatcher-5] [akka://Segmenter/deadLetters] Message [starbucks.bits.segmenter.models.Customer] from Actor[akka://Segmenter/system/sharding/customerActor#-283448532] to Actor[akka://Segmenter/deadLetters] was not delivered. [10] dead letters encountered, no more dead letters will be logged. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.
[WARN] [03/29/2018 19:25:18.422] [Segmenter-akka.actor.default-dispatcher-36] [akka.tcp://Segmenter@10.244.125.8:2551/system/sharding/customerActor] ShardRegion for [customerActor] is using [100.0 %] of its buffer capacity. The coordinator might not be available. You might want to check cluster membership status.
[WARN] [03/29/2018 19:25:18.422] [Segmenter-akka.actor.default-dispatcher-36] [akka.tcp://Segmenter@10.244.125.8:2551/system/sharding/customerActor] Buffer is full, dropping message for shard [325]
[WARN] [03/29/2018 19:25:19.269] [Segmenter-akka.actor.default-dispatcher-38] [akka.tcp://Segmenter@10.244.125.8:2551/system/sharding/customerActor] ShardRegion for [customerActor] is using [100.0 %] of its buffer capacity. The coordinator might not be available. You might want to check cluster membership status.
[WARN] [03/29/2018 19:25:19.274] [Segmenter-akka.actor.default-dispatcher-38] [akka.tcp://Segmenter@10.244.125.8:2551/system/sharding/customerActor] Buffer is full, dropping message for shard [163]
[WARN] [03/29/2018 19:25:20.463] [Segmenter-akka.actor.default-dispatcher-6] [akka.tcp://Segmenter@10.244.125.8:2551/system/sharding/customerActor] ShardRegion for [customerActor] is using [100.0 %] of its buffer capacity. The coordinator might not be available. You might want to check cluster membership status.
[WARN] [03/29/2018 19:25:20.463] [Segmenter-akka.actor.default-dispatcher-6] [akka.tcp://Segmenter@10.244.125.8:2551/system/sharding/customerActor] Buffer is full, dropping message for shard [319]
[WARN] [03/29/2018 19:25:21.242] [Segmenter-akka.actor.default-dispatcher-2] [akka.tcp://Segmenter@10.244.125.8:2551/system/sharding/customerActor] Buffer is full, dropping message for shard [99]
[WARN] [03/29/2018 19:25:21.355] [Segmenter-akka.actor.default-dispatcher-5] [akka.tcp://Segmenter@10.244.125.8:2551/system/sharding/customerActor] ShardRegion for [customerActor] is using [100.0 %] of its buffer capacity. The coordinator might not be available. You might want to check cluster membership status.
[WARN] [03/29/2018 19:25:21.355] [Segmenter-akka.actor.default-dispatcher-5] [akka.tcp://Segmenter@10.244.125.8:2551/system/sharding/customerActor] Buffer is full, dropping message for shard [301]
[WARN] [03/29/2018 19:25:21.358] [Segmenter-akka.actor.default-dispatcher-38] [akka.tcp://Segmenter@10.244.125.8:2551/system/sharding/customerActor] Buffer is full, dropping message for shard [100]
[WARN] [03/29/2018 19:25:21.360] [Segmenter-akka.actor.default-dispatcher-5] [akka.tcp://Segmenter@10.244.125.8:2551/system/sharding/customerActor] Buffer is full, dropping message for shard [107]
[WARN] [03/29/2018 19:25:22.619] [Segmenter-akka.actor.default-dispatcher-5] [akka.tcp://Segmenter@10.244.125.8:2551/system/sharding/customerActor] ShardRegion for [customerActor] is using [100.0 %] of its buffer capacity. The coordinator might not be available. You might want to check cluster membership status.
[WARN] [03/29/2018 19:25:22.619] [Segmenter-akka.actor.default-dispatcher-5] [akka.tcp://Segmenter@10.244.125.8:2551/system/sharding/customerActor] Buffer is full, dropping message for shard [3]
[WARN] [03/29/2018 19:25:23.453] [Segmenter-akka.actor.default-dispatcher-3] [akka.tcp://Segmenter@10.244.125.8:2551/system/sharding/customerActor] ShardRegion for [customerActor] is using [100.0 %] of its buffer capacity. The coordinator might not be available. You might want to check cluster membership status.
[WARN] [03/29/2018 19:25:23.453] [Segmenter-akka.actor.default-dispatcher-3] [akka.tcp://Segmenter@10.244.125.8:2551/system/sharding/customerActor] Buffer is full, dropping message for shard [338]
[WARN] [03/29/2018 19:25:27.131] [Segmenter-akka.actor.default-dispatcher-36] [akka.tcp://Segmenter@10.244.125.8:2551/system/sharding/customerActor] ShardRegion for [customerActor] is using [100.0 %] of its buffer capacity. The coordinator might not be available. You might want to check cluster membership status.
[WARN] [03/29/2018 19:25:27.131] [Segmenter-akka.actor.default-dispatcher-36] [akka.tcp://Segmenter@10.244.125.8:2551/system/sharding/customerActor] Buffer is full, dropping message for shard [30]
[WARN] [03/29/2018 19:25:27.724] [Segmenter-akka.actor.default-dispatcher-37] [akka.tcp://Segmenter@10.244.125.8:2551/system/sharding/customerActor] ShardRegion for [customerActor] is using [100.0 %] of its buffer capacity. The coordinator might not be available. You might want to check cluster membership status.
[WARN] [03/29/2018 19:25:27.724] [Segmenter-akka.actor.default-dispatcher-37] [akka.tcp://Segmenter@10.244.125.8:2551/system/sharding/customerActor] Buffer is full, dropping message for shard [290]
[WARN] [03/29/2018 19:25:27.795] [Segmenter-akka.actor.default-dispatcher-4] [akka.tcp://Segmenter@10.244.125.8:2551/system/sharding/customerActor] Buffer is full, dropping message for shard [254]
[WARN] [03/29/2018 19:25:27.809] [Segmenter-akka.actor.default-dispatcher-36] [akka.tcp://Segmenter@10.244.125.8:2551/system/sharding/customerActor] Buffer is full, dropping message for shard [56]
[WARN] [03/29/2018 19:25:27.828] [Segmenter-akka.actor.default-dispatcher-2] [akka.tcp://Segmenter@10.244.125.8:2551/system/sharding/customerActor] Buffer is full, dropping message for shard [257]
[WARN] [03/29/2018 19:25:27.833] [Segmenter-akka.actor.default-dispatcher-4] [akka.tcp://Segmenter@10.244.125.8:2551/system/sharding/customerActor] Buffer is full, dropping message for shard [163]
[WARN] [03/29/2018 19:25:27.880] [Segmenter-akka.actor.default-dispatcher-4] [akka.tcp://Segmenter@10.244.125.8:2551/system/sharding/customerActor] Buffer is full, dropping message for shard [112]
[WARN] [03/29/2018 19:25:27.893] [Segmenter-akka.actor.default-dispatcher-36] [akka.tcp://Segmenter@10.244.125.8:2551/system/sharding/customerActor] Buffer is full, dropping message for shard [326]
[WARN] [03/29/2018 19:25:27.894] [Segmenter-akka.actor.default-dispatcher-5] [akka.tcp://Segmenter@10.244.125.8:2551/system/sharding/customerActor] Buffer is full, dropping message for shard [146]
[WARN] [03/29/2018 19:25:27.898] [Segmenter-akka.actor.default-dispatcher-3] [akka.tcp://Segmenter@10.244.125.8:2551/system/sharding/customerActor] Buffer is full, dropping message for shard [194]