So I was having a closer look at this and I think there are 2 different issues here:
- single core server results
- multi core server results
Single core server
Single core results are bad, because there are a lot of failure responses from the server:
Status code distribution:
[OK] 68817 responses
[Unavailable] 220868 responses
[DeadlineExceeded] 293 responses
Error distribution:
[219868] rpc error: code = Unavailable desc = all SubConns are in TransientFailure, latest connection error: timed out waiting for server handshake
[227] rpc error: code = DeadlineExceeded desc = context deadline exceeded
[66] rpc error: code = DeadlineExceeded desc = latest connection error: timed out waiting for server handshake
[1000] rpc error: code = Unavailable desc = transport is closing
while for the java benchmark I see
Status code distribution:
[OK] 1523989 responses
[Unavailable] 1000 responses
Error distribution:
[1000] rpc error: code = Unavailable desc = transport is closing
and I see the following errors in the server logs
22:37:36.470 [GreeterServer-akka.actor.default-dispatcher-13] ERROR akka.actor.ActorSystemImpl - Unhandled error: [Stream with ID [27] was closed by peer with code CANCEL(0x08)].
akka.http.scaladsl.model.http2.PeerClosedStreamException: Stream with ID [27] was closed by peer with code CANCEL(0x08)
however after increasing the parallelism setting this errors is gone but the results are not much better. That said I suspect the problem is that the dispatcher settings are not good for a single core server…
3 cores server
Here I no longer see errors
Status code distribution:
[OK] 186355 responses
[Unavailable] 966 responses
Error distribution:
[966] rpc error: code = Unavailable desc = transport is closing
but there is a lot of GC activity, which I see when running with JFR:
and checking the TLAB Allocations tab:
akka.http.scaladsl.model.Uri$.parseHttp2AuthorityPseudoHeader(ParserInput, Charset, Uri$ParsingMode)
seems to be a bottleneck. UriParser was already reported as a bottleneck in Allocation hotspots in akka-http-client · Issue #2739 · akka/akka-http · GitHub
Future steps
I hope these findings can help improve the performance of akka-http and akka-grpc. Meanwhile I will try to contribute some jmh tests to akka-http and If my skills allow contribute some improvements.
Meanwhile if someone has any tips about the single core problems that would be helpful…