we are using play-framework 2.6.17 version with scala version 2.11 for our service.
We notice the heap usage increases gradually over a period of time and from the analysis of heap dump,
the number of RepointableActorRef instances were around 105K with retained heap size of over 200MB.
This in turn cause the fullGC cycle to be triggered frequently.
GC root of these instances point to supervisor(StreamSupervisor-0) of akka.stream.impl.PhasedFusingActorMaterializer instance for AkkaHttpServer.
We looked at the associated ActorGraphInterpreter instances which has the ‘interpreterCompleted’ value as ‘true’ with no subcribersPending. From our code understanding, the RepointableActorRef should be removed from the childrenContainer once the interpreter execution is completed. But in this case,
We notice following sequence of action happening
as part of preStart() method in ActorGraphInterpreter, it checks for ‘activeInterpreters’ which is empty and stops the actor.
But during this time, actor still has ‘UnstartedActorCell’ which receives the stop message and swapping of active ActorCell has not happened.
Now supervisor which added this actorRef as child never gets DeathWatchNotification since the stop message is handled by UnstartedActorCell.
This seems to be a kind of race condition due to which the RepointableActorRef references are retained in the childrenContainer of supervisor forever.
Could someone give pointers on if this issue is already resolved in later version.
I modified the sample play-scala starter project with using Accumulator.flatten api which creates a graph ‘Source. fromPublisher (publisher).toMat(accumulator.toSink)(Keep. right ).run()’ as part of futureToSink method call in FlattenedAccumulator initialization.
When hitting the endpoint ‘http://localhost:9000/count’ at 1000 req in a minute, I can see the following logs,
I can’t reproduce a memory leak with the sample project, putting load on the sample project in prod mode, whenever a gc is triggered I see heap usage drop down to 10-20mb. So there must be some more circumstance causing this for you.
In Akka 2.6.0+ we have eliminated the use of RepointableActorRef for the system level materializers, so it could be worth checking out the play 2.8 release candidate and see if you can still repeat the problem.