We are experiencing something weird on Akka streams implementation. When so many incoming messages are there(after creating a set of actors simultaneously), we get full GC issues. We have analyzed backend using MAT and VisualVM. It complains that we have lot of immutable scala lists(Top dominators seen in MAT) connected to BroadcasHub is remaining in the heap, which is taking much space. When actors are killed, still MergeHub, BroadcastHub and AbstractNodeQueue and not cleaning in the heap area. They are keeping growing more and more when we instantiate more actors. But Flow is getting cleared though under this situation also.
Not sure if that explains your leak, maybe: You must not call context.stop from inside a future callback, it’s actor internal state so should not be touched from arbitrary threads. Instead pipe the future completion back to the actor as a message using pipeTo and make the actor stop when receiving that message.
That is a bug in that sample, I’d recommend that you start with changing it into a stop message and see if the leak is still there after that.
That kind of dynamic fan-in/out is exactly what the merge/broadcast-hub are for, the alternative I can think of would be to make a lower level solution using Source.actorRef and Sink.actorRef and make the dynamic routing with regular actor messaging.
When we locally checked this using VisualVm heap dumps, we saw that heap is not getting filled as previous.
And this was deployed also. Currently it’s under monitoring. So far, we did not get any Full GC issues. We have checked production heap dumps also using MAT. The heap size itself has been drastically reduced! Previously it was more than 2GB(since service Xmx = 2GB, heap grows until that). But now the max heap size we saw is around 600MB! And now these heap dumps does not contain LocalActorRef / BroadcastHub / Scala.Immutable Lists objects which was causing the memory leak…