we are using play-framework 2.6.17 version with scala version 2.11 for our service.
We notice the heap usage increases gradually over a period of time and from the analysis of heap dump,
the number of RepointableActorRef instances were around 105K with retained heap size of over 200MB.
This in turn cause the fullGC cycle to be triggered frequently.
GC root of these instances point to supervisor(StreamSupervisor-0) of akka.stream.impl.PhasedFusingActorMaterializer instance for AkkaHttpServer.
We looked at the associated ActorGraphInterpreter instances which has the ‘interpreterCompleted’ value as ‘true’ with no subcribersPending. From our code understanding, the RepointableActorRef should be removed from the childrenContainer once the interpreter execution is completed. But in this case,
We notice following sequence of action happening
- as part of preStart() method in ActorGraphInterpreter, it checks for ‘activeInterpreters’ which is empty and stops the actor.
- But during this time, actor still has ‘UnstartedActorCell’ which receives the stop message and swapping of active ActorCell has not happened.
- Now supervisor which added this actorRef as child never gets DeathWatchNotification since the stop message is handled by UnstartedActorCell.
This seems to be a kind of race condition due to which the RepointableActorRef references are retained in the childrenContainer of supervisor forever.
Could someone give pointers on if this issue is already resolved in later version.