Hi guys,
I’ve got the following problem. I’ve got a network of ~100 Actors, all running locally. The code is more or less:
There is one Actor “network” and 100 “nodes” network sends step to all the nodes
upon receiving step node sends data to it’s neighbours
upon receiving data node does some calculation on it, and adds the sender to seen set. If all the neighbours are seen it sends ready to network
upon receiving ready network adds node to seen. If all are seen new step is sent.
My problem: while the code runs and with debug enabled I can see a lot of messages is being sent, the JVM process utilizes between 0.5 and 1% of my CPU. Memory should not be a problem as well as the JVM consumes 700-900MB of Ram and Xmxx is set to 4G. There are no disk writes. What is my bottleneck? Why won’t my code use more CPU to run faster?
I’m new to Akka but my suspicion is that the dispatcher might tick not frequently enough? Could that be a thing?
Not really: the dispatcher threads should immediately continue processing further work when there is any available. Are you doing any blocking calls as part of the actor message processing? Perhaps you can share the project somewhere?
It can be viewed on my repo - beware though, as I’m new to Akka this is probably as hot garbage as it gets
I’m currently running ACDemo class from Simulator.scala which is my entry point
One thing that could be problematic is the Await.result in Network.scala: that could lead to trouble later, since it blocks the dispatcher and may lead to dispatcher exhaustion. That might not be so easy to refactor. An important question here is: do these nodes really need to be actors, or could they be ‘regular’ objects inside the Network actor? To determine that I guess the question is if there’s ever another actor than the network that’s talking to the nodes - for example do they ever directly talk to each other?
When you run into this, you’d expect taking a jstack stack trace of your application would show a bunch of dispatcher threads in the Await.result function, though, and I don’t think I see that, so there might be something else going on first. Without diving into the code further I don’t think I see what is causing the lack of progress here, though.
Good lead - I was refactoring Awaits out earlier, must have left this one in by accident. Will work on it soon.
The Actors do talk to each other - when receiving step from network they send data to each other an process it. Will try getting rid of Await and report back
Reporting back
I did pushed the Await to a part of code that’s within a Future[Unit] that is called at the very end of my program and is called in “fire and forget” fashion, but I still don’t get CPU usages higher than 1%.
Any other ideas? Suggestions as how I could try to find the bottleneck? (I’m assuming one exists, because if CPU is running almost IDLE it means something else must be stopping the execution)
One thing I was able to find in Java Mission Control: the Peak Live Thread Count is 54, but most of the time 13-14 threads are being used, and I can’t figure out when did the peak happen (see attached screen shot).
My machine is a 8core/16threads one.
I need to figure out why they are waiting though - the code executed shouldn’t be blocking - the logic is entirely driven by firing messages - an actor receives a message, computes, fires a message. The only thing I can think of is the synchronization being done by checking if all reported - there will be moments, where a lot of nodes in the network will be done and have sent all reported, but those won’t receive any messages then, so shouldn’t affect the threads, right?
