Akka Dispatcher threads with no stacktrace

Play Framework
, ,
1

Hello everyone,

We use Play Framework therefore I’m creating the topic in this forum, but the question is rather related to akka dispatcher, though in the Play’s pool, so I’m not sure if it would be more appropriate to create it in the Akka forum. Anyway, I apologize in advance, and will be happy to move (if possible) or recreate the topic there if need be.

So today we mentioned our Play servers consuming all CPUs, which is unusual because typically the load is around 30-50%. Most of the requests started hanging, though some did work through. We tried to take a thread dump and ran into a strange thing: lots of application-akka.actor.default-dispatcher-N threads were missing the stacktrace, as if they were some kind of OS threads or something (as far as I found while googling, e.g. https://stackoverflow.com/a/14286146/1813669). Here’s a raw thread dump text file: stack.out. I imported it into jstack[dot]review website to analyze, and it showed that there are 1273 non-java threads, of which most are application-akka.* ones. For comparison, I took another thread (link) dump after the Play server restart and there were only 17 of them and the number not growing.

So I’m wondering if anyone has an idea what those empty stacktraces could mean. Would be very thankful for any assistance!

2

I found this topic at Google Forums, which blames the behaviour of Java’s ForkJoin pool: https://groups.google.com/forum/#!topic/akka-user/ES9aybyUHek
We indeed use a lot of blocking operations, but as far as I understand, we do not use the ForkJoinPool because here’s our settings in application.conf:

akka {
  actor {
    default-dispatcher {
      executor = "thread-pool-executor"
      throughput = 1
      thread-pool-executor {
        fixed-pool-size = 1000
      }
    }
  }
}

And it seems to be effective, since I don’t see any 1000+ thread number in the thread dump.