This is on Akka 2.6.8:
Our clustered Akka deployment has become quite unreliable and it seems that we are not receiving all terminated events from remote actors. Today I have found a pretty clear case in our log files.
- Node A (100.64.4.38:2551) gets removed from the cluster
- Node B gets notified about this:
INFO akka.remote.artery.Association - Association to [akka://ClusterSystem@100.64.4.38:2551] having UID  has been stopped. All messages to this UID will be delivered to dead letters. Reason: ActorSystem terminated
- Node B starts watching an Actor on node A, but never receives a termination event. Therefore it assumes the actor is still there. Since I retry sending the message until I either get an acknowledge or a termination event, the functionality is broken.
Is there a bug in AKKA? Or is my understanding of AKKA incorrect that I would ALWAYS get a termination event, even if I start watching after the Actor has died or the node the actor was running on was removed from the cluster?