Akka persistence - remove messages from journal (or mark as confirmed), when recieve confirmation of message delivery

Akka noob here.

So for eg: I have actor X and Y.

Actor X persists message to journal, then it sends message to Y. Y receives message, and sends confirmation back to X to let it know have received message. When X receives this confirmation, I want it to a) delete the message from the journal so the message isn’t replayed on recovery. (this part doesn’t seem to be possible… edit: although, I have found EventEnvelope which does provide sequenceNr, so I could use this to delete from journal actually… but is it a good way?.

b) “Mark” the message as completed (delivered). This part I think will be done with either the Log (using the log on recovery), or through adding “tags” to journal (through event adapter, but I am not sure if that’s possible will update if its a viable option).

This makes me realize, how does akka persistence actually work. If a actor is persisting all messages, and then the actor fails and needs to recover, will it not recover all these messages regardless of delivery? I know it is to maintain state (so for fsm I get it), but if I have a supervisor actor that persists messages to then pass on to workers, surely I would want to be able to change this journals entries so that I wont recover (and then resend) messages that have already been processed? (so thats why I am asking, I am obviously missing something).

Any links, papers, or articles would be really appreciated :)

A side question is this: Each cluster node of mine is isolated, and running on its own jvm. So, message delivery will be prity (highly) reliable. Would it be best to use atLeastOnceDelivery? I feel I mite be being pessemistic in not wanting to use it, as I tested with my code and it sends 106 messages per 100. 6% extra messages seems prity low, but I am trying to build something that will scale (and most the stuff I read seems to say “you dont need it” when it comes to atLeastOnceDelivery). In relation to the main question above, is what I am looking for actualy just atLeastOnceDelivery (as that relies on confirmation aswell)? and is atLeastOnceDelivery actually that “bad” (because the docs and paper seem to say at most once delivery is enough, and its the application (code) that should guarantee delivery.

Akka Persistence, event sourcing, is based on an append only journal so you will never update events that have been stored. I think that answers that b) is not an option. Tagging is not a change of already stored events. You can tag events when they are stored, and later query for events with a given tag.

For the messaging from X to Y you would.

  1. persist the fact that you intend to send the message to Y
  2. when acknowledgment from Y you persist the fact that it was delivered
  3. you need a correlation id between above two events, typically a monotonically increasing sequence number
  4. when recovering the state of X it will replay the events, maintaining those correlation ids in a Set, adding to the Set for 1) events and removing from the Set for 2) events
  5. when the recovery has been completed the Set will contain the ids that were intended to be sent but was never acknowledged (and might need resending)

As a performance optimization you can store snapshots, and delete events up to the snapshot, but that is only an optimization.

AtLeastOnceDelivery is a tool to support this process.

A while ago I wrote an example of how this can be implemented in application code with the new Actor APIs in Akka 2.6 (Akka Typed).

I’m also working on support for reliable delivery for Akka Typed, and you can follow progress in PR #28155.