Memory Leak with millions of HostConnectionPoolStage Slot instances

Akka Akka HTTP
,
1

We have a service that makes outgoing http requests using
Http().singleRequest(request)

We are experiencing memory leak with constantly increasing heap usage, retained, and a heap dump shows more than 14 million instances of akka.http.impl.engine.client.pool.NewHostConnectionPool$HostConnectionPoolStage$$anon$1$Slot

There are also 14,000+ instances of
akka.http.scaladsl.HttpsConnectionContext
indicating that somehow the connections are not freed.

Our service has probably made only a few hundred thousands http requests total, at most, and all to the same host.

Any advice how to avoid the memory leak? (or connection leak)

Akka config:

akka {
  http {
    host-connection-pool {
      max-connections = 1024
      max-open-requests = 2048
      idle-timeout = 600 s
    }
    client {
      idle-timeout = 8 s
      connecting-timeout = 8s
    }
  }
}

We are using Scala 2.12, Akka 2.6.4, Akka Http 10.1.12

Previously we were on Akka 2.5.16, Akka Http 10.1.5, and we didn’t have this issue.

2

Hi @lee,

This probably explains it. HttpConnectionContext is an object that you create and pass to Http().singleRequest, if you pass different instances with every call then a new pool will be created for every different instance of HttpConnectionContext. These pools stay alive for at least akka.http.host-connection-pool.idle-timeout, so if you create more than one pool (request?) per minute they will start to accumulate.

Make sure you only create HttpConnectionContexts once at the start of an app and reuse them throughout your application. Also, because creating an HttpConnectionContext is somewhat expensive in itself.

Hope that helps
Johannes