Given
- A sharded persistent actor, for example a Customer entity
- At some point in time, Customer is in a logical end state and decides to kill itself to free up resources.
- When using akka cluster sharding, it is not advisable for an actor to simply call stop(self()) or send PoisonPill to itself. After all, the actor that represents the Shard Region must be aware that the reference it holds to the entity - Customer in our example - has become stale. Should another message arrive via the shard region for that customer entity which has called stop(self()) or has sent itself a PoisonPill, it will always be forwarded to the Dead Letter Actor because the Shard Region Actor is trying to forward this incoming message to an ActorRef that does not exist any more.
- Akka cluster sharding has a solution for this: passivation messages (https://doc.akka.io/docs/akka/current/cluster-sharding.html#passivation)
- When a Persistent Actor has reached the end of its lifecycle, it should send a Passivate to the parent. The Passivate message is essentially a wrapper around a message that will be sent back by the Shard Region to the Persistent Actor once the shard has marked the ActorRef of the Actor that sent the Passivate as stale (and it will also buffer newly incoming messages for that entity). In most examples online, the message wrapped in the Passivate is a PoisonPill.
- However, when reading the documentation of akka persistence, it warns not to use PoisonPill to kill Persistent Actors, since it could result in lost messages and lost persist-callback-blocks --> akka persistence maintains an internal stash of all the pending persist-callback-blocks and all messages in the inbox. Since PoisonPill is an internal message that is automatically processed (AutoReceivedMessage) , it is possible that it is processed before all persist-callback-blocks are executed --> see https://doc.akka.io/docs/akka/current/persistence.html#deferring-actions-until-preceding-persist-handlers-have-executed
- The suggested solution/workaround is to use a custom “kill” message.
Now the question:
- How does this work in combination with an akka.pattern.BackoffSupervisor (https://doc.akka.io/docs/akka/current/persistence.html#failures). It is an Actor that is in the supervision chain between the shard region and the entity: User Guardian Actor --> Shard Region Actor --> BackoffActor --> PersistentEntityActor
- Assuming a BackoffActor will not recognize a Passivated custom kill message, the Shard Region will simply extract the Passivated message and send the custom kill message back to the BackoffActor, which will then forward it to its child (the customer entity in our case).
- This means that the BackoffActor will remain in memory, even when the entity it is managing has been killed off. How do we remove the BackoffActor from memory as well? How does it respond to a PoisonPill?
- The problem becomes even more complicated when there are even more decorating actors in the inheritance chain: Root Guardian Actor -> Shard Region Actor --> Backoff Actor --> PersistentSupervisingEntityActor --> PersistentEntityActor
One of the only solutions I see is to make the child kill the parent actor --> when the custom-entity-actor receives the custom-kill-message that was wrapped in the passivate, it knows that it is “decorated” with a backoff actor and it will kill the parent / backoff actor.
The downside of this approach is that it comes with tight coupling.
Any suggestions or thoughts are greatly appreciated