We are facing issues in Play Framework 2.7.x with a use case where we try to serve a large number of concurrent clients with an SSE event stream.
We started to see the issue after upgrading from Play 2.4.x to Play 2.7.x where the major change introduced was the introduction of Akka HTTP as the default HTTP server and the replacement of Iteratees with Akka streams for feeding the Play EvenSource
As for the stream implementation (the alternative for the Iteratees implementation we had before), we have already tried out 2 different implementations
- using Akka Streams BroadcastHub
- using plain actors (based on the reference implementations used by the LinkedIn team)
We believe at the moment that both implementations of the stream are not the issue, BUT that the problems are introduced by default limits in Akka HTTP server which we appear not to be able to tweak to our advantage.
We have noticed that by default the Akka HTTP server has the following values
akka.http.server.max-connections = 1024
akka.http.server.backlog = 100
I am mentioning the above settings as in a 2016 blog by some LinkedIn engineers (https://engineering.linkedin.com/blog/2016/10/instant-messaging-at-linkedin--scaling-to-hundreds-of-thousands-) , the latter one was explicitly mentioned as a critical parameter for scaling up.
we are running 2 instances of Play in our test environment. From that we conclude that with the default settings there exists some limit of 2048 concurrent connections. Obviously we would like to scale up dramatically on a single node to accomodate for a much higher number of users, just as LinkedIn is doing.
The LinkedIn blog did not have to mention the max-connections, because this article was written at a time when Netty was still the default HTTP server underneath Play.
Today however, Akka HTTP has become the default HTTP server, and it appears to have a default value for max-connections of only 1024.
So, we decided to increase this default value, but somehow we have the impression that any attempt to apply this change either is not applied, or somehow there is still some other hidden limitation that prevents us from scaling up.
In the Play documentation, we find that only a subset of the default AKKA http parameters can be altered (https://www.playframework.com/documentation/2.7.x/SettingsAkkaHttp). To our surprise, the max-connections parameter is not listed by the Play configuration. We tried to set the parameters as outlined below
This does not appear to have any effect.
A parallel change we applied was to also add these parameters in the play.akka.config section we are using.
play.akka.config = “play-akka”
Unfortunately we are still seeing our HTTP server come to a halt once our test program exceeds the supposed max limit based on the default settings (2 x 1024 = 2048).
When we run the same load test on our old software (with Netty underneath), everything works fine.
We are scratching our heads right now with the following questions
- Are we doing this correctly ?
- Can someone tell us from their own experience whether there are limitations on the Akka HTTP server that we cannot control ? Is Netty the only valid option for this kind of use case ?
- Did we miss something else ?
Thanks in advance for giving this a thought.