Looking for some advise regarding Akka Persistence and ensuring atomic command operations where they involve more than one persistent actor.
The trick is handling error scenarios where one persistent actor successfully persists but another does not (leaving an inconsistent state).
My current idea to solve this is for the process manager actor (responsible for calling the persistent actors involved) to request the current Sequence Number from each persistent actor prior to calling persist on each of them.
If any problems occur mid process then the process manager would force the persistent actors involved to restart themselves and use the ‘Recover(toSequenceNr = X)’ to replay all the events up to the point prior to the whole process starting.
What I’m stuck on is how on earth to ask a persistent actor to restart itself to a given Sequence Number.
To my understanding causing an actor to restart will replace the backing instance of that actor, so passing the toSequenceNr value to the actor in the restart message is useless (as the value will be lost upon restart).
The only way I can think of to pass the toSequenceNr value to a persistence actor is for the persistent actor’s parent to stop the child completely and then recreate it while passing the toSequenceNr as a props parameter value.
This seems like a bit of overkill to me (pardon the pun) so I’m wondering if there is another way to restart a persistent actor and have it recover to a given sequence number?
Or am I barking up the wrong tree entirely trying to solve the atomic issue this way?
My personal take on this is that what you’re trying to achieve is basically mismatching with common wisdom on doing transactional processing in DDD and Event-sourcing, which applies to persistent actors too, when the actor represents some business logic.
When you approach the business modelling, you’re advised to create what is called an Aggregate entity, a data structure that represents a common model root enitity and all his dependant sub-entitites and values.
To get the best outcome, you should define the aggregate such that it matches with a transactional boundary, i.e. a domain operation on the aggregate should be able to complete without interacting with any other aggregate, and it should succeed of fail, atomically.
Thus you recover the transactional semantic at the aggregate level.
It’s also common that some operations should necessarily span more than one aggregate, and in that case you define a Process or Saga (in DDD terms), which consists essentially in defining compensating actions for each operation you’re applying.
E.g. if you have aggreagates (actors) Alice and Bob and you need for Alice to process event Received Invitation, and then Alice should ask Bob if he can come as companion before replying to the RSVP, then
apply the event to Alice
ask Bob for availability
based on Bob’s reply, answer to the RSVP with a yes or no, specifying the companion
If 2 fails, you want to run a compensating action that reverts the state of Alice to what it was before 1 happened.
This would mean that you can retry the whole process from start.
The details of how to do this can be different, but essentially this is the gist of it.
I encourage you to read something about ProcessManagers and Sagas and Aggregates in DDD-terms, so you can get some advice on how this is commonly handled.
From there you can choose how to proceed, but with a clearer picture of the whole situation.