I have contributed a new tech stack variant to TechEmpower performance benchmarks based on Akka HTTP and Slick.
Now there is a problem. Locally the test results look fine, but on the servers most of the test fail to perform in any reasonable way, to be seen in the results here:
I am a of course worried that I may have made a basic mistake in the code. However, as it seems all fine in local runs on my laptop, I do not know where to start looking.
Thus I now come here to ask the Akka Team and perhaps also Slick experts for a review.
I looked into it a while ago but there are no obvious mistakes. The most likely culprit would be the docker image / JVM configuration. Might make sense to print out Runtime.availableProcessors() to see if the JVM can figure up the right amount of processors from the docker runtime configuration.
the new results are out. The logs of how many processors the JVM determines on that server I cannot see. Locally it worked correctly. However, we can derive, from the fact that only akka-http-slick-postgres has the problems and akka-http works correctly, that this probably works, also since I use the same Dispatcher configuration.
I discovered that it is quite interesting to look at the individual data tables where the different performance with respect to the amount of concurrency can be seen.
For example, the use case “JSON serialization” degrades harshly with more concurrency, while the use case “Single query” is capped early at a max of 50000 requests.
So probably some implementation mistakes are lurking and they are potentially different from use case to use case.
Maybe someone can review the Dispatcher(s) and Executors used in my code.
It could well be that I have made a mistake by either having a blocking operation on the main Dispatcher or by having introduced a custom undersized thread-pool.
Slick also introduces its own thread pool, which must be sized correctly with respect to the available resources and also the size of the connection pool:
@TimMoore Thank you, this I was aware of though. I actually tried different sizes of the HikariCP already. I also verified that the pool actually gets used and configured. But someone helping in the investigation or reviewing the code could have another look, would be appreciated.
Note however that even use cases “JSON serialization” and “Plain text” perform poorly. They do not even make use of Slick/Database.