Dropping buffered messages to shard during hand off to avoid re-ordering

Hi folks,

I’m having the following issue during deployments. It seems that when rebalancing shards, some of the messages for that particular shard are been dropped and because of that, never answered.

WARN akka.cluster.sharding.ShardRegion CategoryTreeSupervisor: Dropping [89] buffered messages to shard [14] during hand off to avoid re-ordering

Do we have any idea why this is happening or how can I fix it?

Message delivery isn’t guaranteed so there can be cases where they will be dropped. It should be a good best effort though, so if you have a small example application that we can try for reproducing the problem it would be interesting to look into.

It’s important that it’s using graceful leaving when performing the rolling updates. That should be the default via the jvm exit but would be good to confirm. Should be visible in logs. No downing.

Make sure you use lates Akka 2.6.x patch version, and enable new allocation strategy. Cluster Sharding • Akka Documentation

Also, look into using app-version property. Rolling Updates • Akka Documentation