Can I use Akka Mailbox to control race condition in a frontend?
Generally we have 100,000 TPS(message per second) need to handle. Each message belong to one user. Sometime two messages belonging to the same user come in at nearly the same time. So we need to control the race condition for such case by processing them one by one.
I thought Akka Mailbox can handle it, and I also tried in simple spring application with local service. it looks works. But not quite sure it can work in production in k8s. My questions are:
1, First to confirm the Akka Mailbox can do this work. My solution is to create Actor for each user, the actor name is the username. and then follow actor usage to send message to remote actor which is the service instance in a Akka cluster. I think Akka will follow the mailbox mechinism to queue the message until the previous message got processed. Is this correct for the solution?
2, If we have many users, e.g. 1M concurrently. That means the actors are 1M as well. If we find the frontend instance is busy or reach the threshold, can we scale out the frontend instances and how the Akka to control the same user actor consistence if the same user messages goes into different frontend instances. I know there is Akka cluster, but looks the cluster means backend services cluster in our case.
If each user is represented as a single actor, and you send all messages per user to that actor they will be processed one at a time, so there will be no concurrent access, however if the messages are sent from different places concurrently, the order they are processed by the actor will still be a race.
For scaling this out, there is Akka Cluster Sharding, which manages keeping one actor of each id alive across the nodes of the cluster for you.
About the Akka Cluster, not sure about the case, can you help my understanding below:
1, The external messages can be routed to any node of the cluster.
2, In the node, the user actor will be looked up in the cluster by an unique id, if not found, a new user actor will be created and the message will be put in the mailbox for processing.
3, At nearly the same time, another message of the same user comes in and do the same as step 2. And maybe in a different node. not sure what will happen. I think only one actor can be created and another one will raise exception. If it is correct, the another one can try to lookup again and put the message into the actor for processing.
4, Can the actor have a life time? If an actor have no messages in the mailbox for a period, e.g. 10s. The actor can be cleaned up in order to save cost. The user actor number could be 1M.
5, Can I use normal Cluster instead of Cluster Sharding? not sure a normal cluster must have a Sharding.
Please read the docs for Cluster Sharding which I linked to, perhaps start by first reading the actor introduction and then maybe also the Cluster section.
@johanandren
When will the second message in the mailbox to be kicked off for running?
1, after the first message processed and returned by createReceive() of the receive actor. or
2, after the first message replied by telling the result to the sender.
I experienced the case in createReceive() of the receive actor. We call an async processor to process the message and returned immediately. In the callback of the processor, we tell the result to the sender. In this case, my observed result is above item 1. But I expect only the sender receive the result of first message then could process the second message, that is Item 2.
The actor will not pick the next message from the inbox until it has completed the processing the current one.
However, if you do async work, such as triggering some processing in a Future or CompletionStage, that means that the actor processing from the actor’s point of view is completed. A common way to deal with that is to switch behaviors and use stash to buffer messages until the current async work completes.