I’m upgrading an old application based on Play 2.4 to 2.5, the first of many steps to get it to the latest Play version. After the upgrade, though, I’m noticing increased memory usage, growing over time until the JVM or the host system crashes.
I’ve tried every tool in my box to figure this out, without any luck. The worst part is that this leak from native or otherwise untracked memory. Memory profiling using YourKit shows mostly regular heap and non-heap memory pools, but the resident size of the process as reported by the OS grows until the kernel decides to kill it, or the whole system comes to a crawl. I’ve tried switching the JVM from OpenJDK to OpenJ9, but if this is a VM bug it’s shared by both implementations.
For example. Here the profiler is reporting 2.5G + 494MB allocated for heap and non-heap memory altogether:
And yet, top shows over 6.7G resident size.
It just gets worse over time until the process or the whole system becomes unresponsive.
Has anyone here seen anything like that, or has any advice to help me figure out where that memory usage is coming from?
Environment:
Play 2.5.19 (Akka 2.4.20)
Java 8u212 (tried on both Amazon Corretto and AdoptOpenJDK OpenJ9)
That’s my eventual plan, but getting this app to run on 2.7 won’t be easy. The app has been around for 4 years now, and would require extensive changes. I’m trying to do this incrementally.
Do you see OOM errors or only more memory being allocated? I would also suggest you analyze a heap dump, especially if you are observing OOM errors. Very hard to help you only with the information we have here.
No errors thrown, just a growing RSS as reported by the OS. The allocated memory reported by the profiler is mostly stable, capped at -Xmx (4GB in my specific case). So it’s native memory being leaked, and I have no idea where from.
I realize this post is a very long shot, thank you for even trying to help.
Got it. But which process the OS is reporting as the most memory hungry? Is it the JVM process? What if you set -Xmx to a smaller value (if possible even below 1GB)? Are you using Netty native transport? Also, which kernel version are you using? I know Ubuntu had some 'bad kernel releases not long ago](https://www.spigotmc.org/threads/ubuntu-18-04-bad-kernel-update.357871/).
The JVM process, yes. I’ve tried reducing the max heap size; it helps for a while, but eventually is leaks just as much memory. We’re on Ubuntu 14.04 (Kernel 3.13.0); ancient, I know - upgrading distro and kernel is my next hail.
