No CPU spikes or Ingest Spikes.
Only 10% of messages are getting processed. Rest are “disappearing” /not received to the Actor.
Any pointers ?
You will need to provide a lot more information. You frame the question as “performance degradation” but the symptom you list is 90% message loss. But you really don’t give much information around that either.
I think we’d need to understand a little bit more about your topology (how are these messages getting added: e.g. locally/remotely/from streams/over grpc), what your networking stack looks like (netty/artery-tcp/artery-aeron), what you general workload does, what version of Akka, and why you think this is performance related. Also whether you have Telemetry installed and what you are seeing there. You don’t say a lot about how the messages disappearing, but “disappearing” messages are always going to result in some errors in the logs (either on client or server), so logs would be helpful too.
Without logs/data/information it is very hard to provide anything material, but I will say that I’m skeptical of it being performance degradation. Message drops are generally going to be caused by networking issues and/or misconfiguration. This is a massive oversimplification, but performance degradation is generally going to be evidenced by messages not getting processed in a timely fashion and/or resources getting strained (like memory or db). Message dropping is instead going to be thread starvation, networking problems, misconfiguration, and/or bounded mailboxes.
Also, you phrase your question “why are all message not processing in the queue”. Which is curious because while there are some scenarios where messages fail to get put into the mailbox queue (e.g. network failures, bounded mailboxes) I don’t think there’s any real way that messages will not get processed once they are in the mailbox. If messages are “disappearing” once they are in the mailbox, that’s almost certainly just a problem in your actor’s logic.