Akka-HTTP Request-Level Client-Side API creating too many threads

Akka Akka HTTP
1

Hello!

First, since I’m new here, I can’t post all the evidence (images) for this question. I’ve already asked in Stack Overflow, without getting any answer so far. Further screenshots of the observed behavior can be seen there.

I am seeing an odd behavior in a microservice using Akka-HTTP (10.1.11 / Scala 2.12.9) for consuming external services through the Http().singleRequest method.

The observed behavior is that we’re creating a thread pool per a set of outgoing requests. This TP contains three threads that are created as part of the Akka’s default dispatcher but those are never used (they stands in Park state until their EOL). The TP recycling happens in a non-deterministic way, meaning that some threads are killed after 200.000ms, for example, while the next set of threads belonging to a newly created TP are killed in a lower or upper timeframe (might indicate that it’s not following the akka.http.client.idle-timeout configuration).

This is the overall thread count (created) through a 2-hour span:

866×405

So, my questions here are:

  1. Am I looking in the right direction? I mean, is this indeed a thread pool leak or is this the supposed behavior of Akka-HTTP?
  2. If this is the intended behavior, why is it desirable to spawn new threads per request? Isn’t this too expensive to perform? And why those newly spawned threads keeps unused?
  3. If it’s not the intended behavior, where should I look at?

It’s worth to mention that we’re using the following configuration:

http {
    # https://doc.akka.io/docs/akka-http/current/client-side/pool-overflow.html
    host-connection-pool {
      max-connections = 16
      max-open-requests = 64
      response-entity-subscription-timeout = 10s
      max-connection-lifetime = 60s
    }
  }

Finally, the Request-Level Client-Side API docs shed some light, but I’m not entirely sure about the inner workings of this approach.

Thanks in advance

cc. @jrudolph @ktoso (I mention you both because I know that you might get any lead on this :sweat_smile:)

2

I replied on SO. I think I found a reason for that behavior in the default configuration of the default-blocking-io-dispatcher.

In short, it seems that all threads of the pool are shutdown when the timeout triggers and recreated when a single task comes in.

See https://github.com/akka/akka/issues/29238 for more details. The quick workaround is to set

akka.actor.default-blocking-io-dispatcher.thread-pool-executor {
  allow-core-timeout = off
}