Why is my heap usage going up on an Akka HTTP app without traffic?

I have a basic CRUD and websockets app that uses Akka HTTP with Slick for database connections on top of HikariCP. I noticed that the memory profile grows linearly until it hits the heap max, even though there is zero traffic. Attached is a screenshot from a VisualVM profile. The sharp drops are where I manually performed GC or took a heap dump. The heap dumps are similar so I am not sure what’s going on.

1 Like

Hi @nsadeh,

The term “heap usage” is a bit misleading for a runtime like the JVM that doesn’t track which memory is alive. The number shown is basically how much memory has been allocated since the last garbage collection but not whether the data is still referenced (= in use). Only during a garbage collection the heap is walked and data is identified that is still referenced and all other data is discarded.

As you have seen, the JVM often runs a garbage collection before a heap dump, so that the heap dump only contains alive memory. The graph looks as if the memory used after a GC stays constant, which shows that there’s no memory leak.

So, what you are seeing that in a seemingly idle scenario, there are still some allocations. Akka and Akka HTTP have some infrastructure like the scheduler that run a bit of code even if nothing else is running. It is possible to write (heap) allocation-free code but most Java code does some heap allocation, so that’s probably what you see here.

The “cost” of these allocations is “just” the CPU time of an occasional GC. This is usually nothing to worry about. I would suggest to monitor the total CPU time during idle times and see if it is significant.

Hope that explains it.
Johannes

1 Like

Thanks - but my deployment of this app on Heroku flagged that heap memory exceeded limits. If I just reduce Xmx to under my contract’s memory limit, will I be fine? I wanted to ensure this isn’t a leak or anything. Also, is it normal that Akka allocates memory at 16MB/minute? That is approximately the rate I am seeing.

I would assume that reducing the memory to be under the heroku limits would simply increase the GC rate to keep the promised allocation

Yes, should be fine. Make sure to leave a bit of room (50-x00MB depending on your app) for non-heap parts.

Doesn’t sound too bad. I had a quick try and can confirm that it’s most likely the scheduler that will allocate a bit of memory on every tick. I created Avoid allocations in scheduler while idle · Issue #30093 · akka/akka · GitHub to track that we might be able to do better than this (even if insignificant).