TechEmpower performance benchmark results round 15, including several play2 setups

#1

Not sure if this is known already, since the benchmarks were published February 14, but here it is: https://www.techempower.com/benchmarks/#section=data-r15&hw=ph&test=json

For those not in the know, this is a massive set of benchmarks for a lot of technologies and frameworks. The filter panel allows you to do fine-grained selection of what’s important to you and then you can check the results, which reach over plenty of tests for json serialization, plain text, queries, etc… both on physical hosts and on cloud hosting(Azure).

It’s certainly an interesting set of data as a whole, and more specifically for play2’s place in it.

Beyond posting this as a purely informative thing, I wonder if there’s been communication between the dev team(or Lightbend) and the people behind these benchmarks, in relation to how the setup of the play instances happened.

I’m reminded of this: https://github.com/playframework/playframework/issues/8010

EDIT: Nevermind, it’s all listed on their github.

For instance, play2-java can be viewed here: https://github.com/TechEmpower/FrameworkBenchmarks/tree/master/frameworks/Java/play2-java

EDIT2: I just stumbled upon this: https://github.com/TechEmpower/FrameworkBenchmarks/blob/master/frameworks/Java/play2-java/play2-java-ebean-hikaricp/build.sbt

Where the Netty server is enabled yet the Akka server isn’t disabled, which is stated to be necessary if you’re going to run Netty on 2.6.x: https://www.playframework.com/documentation/2.6.x/NettyServer

To use the Netty server backend you first need to disable the Akka HTTP server and add the Netty server plugin to your project

Not sure if that means Akka is still running the whole time, if there’s performance issues because of that, etc…

Yeah, smarter people than me can probably make a more balanced judgement of this, I suppose. I’ll stop editing the top post now.

(Rich Dougherty) #2

Can you try running one of the projects to see which server is actually running? I think it was @mkurz and @godenji who were kind enough to update the tests in the latest round.

#3

I launched the play2-java project, which sets Netty Server.

Starting it up in dev mode just shows:


--- (Running the application, auto-reloading is enabled) ---

[info] p.c.s.NettyServer - Listening for HTTP on /0:0:0:0:0:0:0:0:9000

(Server started, use Enter to stop and go back to the console...)

Checking the plugins give this list:

	sbt.plugins.JvmPlugin: enabled in root
	sbt.plugins.CorePlugin: enabled in root
	sbt.plugins.JUnitXmlReportPlugin: enabled in root
	sbt.plugins.Giter8TemplatePlugin: enabled in root
	play.sbt.Play: enabled in root
	play.sbt.PlayAkkaHttp2Support
	play.sbt.PlayAkkaHttpServer: enabled in root
	play.sbt.PlayFilters: enabled in root
	play.sbt.PlayJava: enabled in root
	play.sbt.PlayLayoutPlugin: enabled in root
	play.sbt.PlayLogback: enabled in root
	play.sbt.PlayMinimalJava
	play.sbt.PlayNettyServer: enabled in root
	play.sbt.PlayScala
	play.sbt.PlayService
	play.sbt.routes.RoutesCompiler: enabled in root
	play.sbt.test.MediatorWorkaroundPlugin
	play.twirl.sbt.SbtTwirl: enabled in root
	com.typesafe.sbt.SbtNativePackager: enabled in root
	com.typesafe.sbt.packager.archetypes.JavaAppPackaging: enabled in root
	com.typesafe.sbt.packager.archetypes.JavaServerAppPackaging: enabled in root
	com.typesafe.sbt.packager.archetypes.jar.ClasspathJarPlugin
	com.typesafe.sbt.packager.archetypes.jar.LauncherJarPlugin
	com.typesafe.sbt.packager.archetypes.scripts.AshScriptPlugin
	com.typesafe.sbt.packager.archetypes.scripts.BashStartScriptPlugin: enabled in root
	com.typesafe.sbt.packager.archetypes.scripts.BatStartScriptPlugin: enabled in root
	com.typesafe.sbt.packager.archetypes.systemloader.SystemVPlugin
	com.typesafe.sbt.packager.archetypes.systemloader.SystemdPlugin
	com.typesafe.sbt.packager.archetypes.systemloader.SystemloaderPlugin
	com.typesafe.sbt.packager.archetypes.systemloader.UpstartPlugin
	com.typesafe.sbt.packager.debian.DebianDeployPlugin
	com.typesafe.sbt.packager.debian.DebianPlugin: enabled in root
	com.typesafe.sbt.packager.debian.JDebPackaging
	com.typesafe.sbt.packager.docker.DockerPlugin: enabled in root
	com.typesafe.sbt.packager.docker.DockerSpotifyClientPlugin
	com.typesafe.sbt.packager.jdkpackager.JDKPackagerDeployPlugin
	com.typesafe.sbt.packager.jdkpackager.JDKPackagerPlugin
	com.typesafe.sbt.packager.linux.LinuxPlugin: enabled in root
	com.typesafe.sbt.packager.rpm.RpmDeployPlugin
	com.typesafe.sbt.packager.rpm.RpmPlugin: enabled in root
	com.typesafe.sbt.packager.universal.UniversalDeployPlugin
	com.typesafe.sbt.packager.universal.UniversalPlugin: enabled in root
	com.typesafe.sbt.packager.windows.WindowsDeployPlugin
	com.typesafe.sbt.packager.windows.WindowsPlugin: enabled in root
	com.lightbend.sbt.javaagent.JavaAgent
	com.lightbend.sbt.javaagent.JavaAgentPackaging
	com.typesafe.sbt.web.SbtWeb: enabled in root
	com.typesafe.sbt.jse.SbtJsEngine: enabled in root
	com.typesafe.sbt.jse.SbtJsTask: enabled in root

Which shows both Akka and Netty as “enabled in root”.

So, I’m not sure what to make of that.

Adding .disablePlugins(PlayAkkaHttpServer) in the build.sbt produces the exact same plugin list, still saying the akka http server is enabled in root

Is there another way I should be looking at this?

EDIT: Also the content of the zip produced by sbt dist is identical, with or without akk disabled in build.sbt

EDIT2: Also, there seem to be certain steps missing from the dockerfile in the database projects. They all expect a mysql database yet the code contains no setup details, nor do the dockerfiles.

(Rich Dougherty) #4

If you take a thread dump of the running app, take a look for Netty threads. That should confirm whether it’s running.

(Rich Dougherty) #5

They’re probably connecting to TechEmpower’s provided MySQL database. On the TE benchmark site there are descriptions about how each test should behave.

#6

Sorry for the late reply. I ran the project from the jar produced by sbt stage, as per the dockerfile.

I ran it 3 times, each time I called the routes a few times and then did a kill -3 on the PID to force a thread dump.

With Netty set as server provider: https://gist.github.com/KoenDG/b8f44187e148d56804f4f9591c9c88c2

With nothing set as server provider: https://gist.github.com/KoenDG/015b209a04a3afb9828efed22b35390d

With Akka set as server provider: https://gist.github.com/KoenDG/4f7fe24c8f14e55889bdf8adaa3b2b6f

The first 2 look very similar, with the Akka one having some thread names not found in the first two. If someone more knowledgeable in these matters could have a look to verify all is ok, that would be appreciated.

Only real thing of note is that the once started with Netty print this every time a route is called:

[warn] p.c.s.NettyServer - Ignoring unknown Netty channel option: TCP_FASTOPEN
[warn] p.c.s.NettyServer - Valid values can be found at http://netty.io/4.0/api/io/netty/channel/ChannelOption.html and http://netty.io/4.0/api/io/netty/channel/epoll/EpollChannelOption.html```
(Rich Dougherty) #7

I think the netty-event-loop threads show that Netty is running. So it’s running for the Netty server provider, but not the Akka HTTP one, which looks correct so long as you’re supplying -Dplay.server.provider on the command line. If you don’t supply that value then it defaults to the value of play.server.provider in a reference.conf file on the command line.

These values are set inside reference.conf files in the Play akka-http and netty JARs, so you should only have one of these on the classpath, otherwise the server that is run will depend on which JAR is first on the command line. Can you list out the JARs to see if there is more than one server JAR on the classpath?

Looks like TCP_FASTOPEN is an option if you use a native transport.

#8

Alright, back with the result. Sorry for the timeout once again, life’s just been busy.

I did the test in all 6 situations(3 possible provided options times 2 possible compile options: with disabling akka and without).

Short story: if you compile the application without disabling the Akka HTTP server plugin, both the Akka and Netty server jars will be on your classpath, regardless of what you provided as a setting in the commandline.

If you compile the application with .disablePlugins(PlayAkkaHttpServer) only the Netty server jar will be on your classpath and trying to launch with akka specifically won’t work.

The output:

Without disabling Akka:

With disabling Akka:

So yeah, it clearly affects how your application is loaded. Someone who forgets to disable the plugin but runs explicitly with the Netty server provider will still have the Akka Http jar loaded onto their classpath.

TechEmpower benchmark round 16 is out, also with play framework being tested in several categories
#9

@richdougherty Does any further testing need to happen for this?

#10

Gonna ping this again.

@richdougherty @marcospereira

Post from April 25 has the conclusions in it. Does anything further need to happen?

(Marcos Pereira) #11

Hi @KoenDG,

Sorry for taking a little longer to reply here. Your conclusions here is what we expect to happen:

In other words, the Akka HTTP server needs to be explicitly disabled. Since it is the default, and Play needs a server, it is enabled by default. This is a by-design choice, so there is no further work to do here. :slight_smile:

Best.

#12

Sure, but the whole reason I looked into it was that the code that’s being used by the people at FrameworkBenchmarks, specifies Netty as the server in the commandline arguments. And without disabling Akka, that’s also going to be on the classpath.

So, what I’m wondering is: does that mean both HTTP servers are running and one is just not reachable… but still running?

(Marcos Pereira) #13

Oh, sorry. I misunderstood you. I thought you were talking about taking some actions on Play side.

No. It is not possible to run the two servers at the same time since they would try to use the same port. So one of them would fail.

What is happening is that since both are present in the classpath, both reference.conf files from dependencies will be loaded and the last wins, since it will overrides the value of play.server.provider. I suspect the classpath entries are ordered lexicographically, and that is why Netty wins.

Best.

#14

Wow, I can’t understand that I didn’t think of this myself. It’s so utterly logical and I just went by it completely.

Either way, I created a PR over at their github, updating both the Scala and Play projects to 2.6.15: https://github.com/TechEmpower/FrameworkBenchmarks/pull/3849

EDIT: And it got accepted, hooray.