Scaling Akka Websockets, tcp settings?

akka-http
configuration
akka

#1

Hi all,

this will be a far shot but hopefully someone can provide some insights. I’m currently battle testing an Akka based WebSocket server that should handle several 10k connections. The problem is that I hit some limit around 10k connections (addional work per connection increases …). It looks like this could be because one of my cores is maxed out, the others still have capactiy.

A snapshot of my CPU usage (mpstat) looks like this

Average: CPU %usr %nice %sys %iowait %irq %soft %steal %guest %gnice %idle
Average: all 38.87 0.00 6.39 0.00 0.00 2.90 0.26 0.00 0.00 51.58
Average: 0 11.80 0.00 11.06 0.00 0.00 46.48 0.39 0.00 0.00 30.27
Average: 1 39.87 0.00 6.42 0.00 0.00 0.03 0.26 0.00 0.00 53.43
Average: 2 40.04 0.00 6.19 0.00 0.00 0.04 0.26 0.00 0.00 53.47

The 46% soft increase for more demanding tests to 90% and in the end seem to limit how far I can get. My assumption is that this might be a bottleneck in the TCP selector / something close to it, before offloading to normal actors and properly distributing the load.

In the akka-core documentation there is a setting “nr-of-selectors = 1”. I tried to change that but the result is not different, still one core going wild.

  • does the akka core configuration for TCP apply to akka http

  • can I configure how akka websocket servers handle the IO details with Akka TCP settings

  • is it enough to just set 4 in order to get 4 selector threads?

  • any other ideas? Might be the totally wrong setting after all

thanks in advance, best Michael


#2

Update:

I did a further check and the bottelneck appears to be either outside of Akka or Akka has for some reasons a afinity to Core 0 (running the server twice didn’t change the CPU stats, still exactly one core under siege for %soft)

nevertheless before closing this topic if anyone knows:

  • does the TCP configuration affect Akka HTTP?
  • for the selector setting, is it enough to just flip the number of selector threads or do I have to change the dispatcher as well?

thanks, best Michael