in one of my applications I’m using the akka-http client and a few days ago I did some load testing and profiling to see, which components are doing most of the allocations and are slowing down the application. I noticed that around 40% of the allocations are done by the akka-http components. To be able to isolate the problem, I created a simple script which calls a HTTP mock server via Http().singleRequest 100,000 times.
The profiling showed that for these 100,000 calls around 650MB of data was allocated in the
NewHostConnectionPool in the function
runOneTransition (package akka.http.impl.engine.client.pool.NewHostConnectionPool). The reason for the allocations are the
debug function calls in this function (e.g. see https://github.com/akka/akka-http/blob/master/akka-http-core/src/main/scala/akka/http/impl/engine/client/pool/NewHostConnectionPool.scala#L252). These debug calls trigger many char / String allocations even if you are not using the debug-level logging, because the log level check is only executed in the debug function (but the String allocation gets triggered already before).
To avoid these allocations the log level check could be either inlined or a macro based solution could be used. I check it locally by removing the debug calls and it lead to around 50% shorter and less GC calls and improved in general the allocation profile of the application.
After these allocations were removed I saw that there was a second function call which was allocating around 400MB of objects for these 100,000 requests. The reason is this function call: https://github.com/akka/akka-http/blob/master/akka-http-core/src/main/scala/akka/http/impl/engine/client/PoolInterface.scala#L168. This is calling the
UriParser.parseHost() function which is in the end allocating
akka.parboiled2.ValueStack objects. Based on this it looks like that parsing the Host is done very frequently, although I was only calling one host in my test script. I’m not totally into the internals of akka-http, but based on what I understood is that it is using host-connection-pools. Wouldn’t it be possible to parse the host only once per host-connection-pool and reuse that to avoid doing this for every request?
I searched in the open issues and also in the forum, but didn’t found anything so my question in general is if these two hotspots are already known? By improving these two allocation hotspots this would reduce the overall allocations by more than 50% and would improve the overall performance of the client implementation. I wanted to share these findings here.
I have tested this with scala 2.12.8, akka-http 10.1.10 and akka-streams 2.5.25.
Best regards and thanks in advance!
P.S. I wanted to share more references to the actual code, but as a new user I’m only allowed to share two URLs.