How to debug the delay between Netty and Akka Default Dispatcher

Play Framework
1

I’m facing a significant delay(30-90 sec) between the time when Netty receives the request and the time when Akka’s dispatcher starts executing the Controller’s action.

I’m using Play 2.5.2. The Netty is running with default configurations. Akka’s default dispatcher is configured with fork-join-executor having 12 threads on a 4 core machine.

In the thread dump, I’m able to see that almost 10/12 threads are in WAITING (parking) state while other in TIMED_WAITING (parking) state. This is resulting in heavy client timeouts on the servers.

The machine also CPU has ~30-40% utilization. I presume that the free fork-join-executor should immediately pick the messages from Netty.

Follwing are the logs from netty and Play’s logger :

Netty Wire:
2019-06-06 13:10:46.141 +0000 [id: 0xb2584eca, L:/10.140.0.22:9000 - R:/X.X.X.X:X] RECEIVED: DefaultHttpRequest(decodeResult: success, version: HTTP/1.1)

Play’s logger:
2019-06-06 13:11:19,550 [INFO] from application in application-akka.actor.default-dispatcher-10

application.conf:
akka.actor.default-dispatcher.fork-join-executor{
    parallelism-factor = 3.0
    parallelism-max = 64
    parallelism-min = 8
    task-peeking-mode = LIFO
}

What could be the possible reason for the delay here? I expect the delay to be near 0 here.

Thanks