I have akka cluster and one Notification actor, that is cluster singleton and persistent at the same time.
Sometimes cassandra db (persistent journal) connection fails and Notification actor just gets stopped (but not restarted!).
What is the best way to handle its restart after stopping?
I have found one suggestion how to deal with singleton supervising:
https://stackoverflow.com/questions/36701898/how-to-supervise-cluster-singleton-in-akka/36716708#36716708 (just create some mediator actor that will supervise target, also watching for termination and start target actor should be done manually).
What about Backoff pattern? Seems like it is what I need . Backoff supervisor will constantly start my stopped Notification actor automatically after some delay.
val supervisorProps = BackoffSupervisor.props( BackoffOpts.onStop( notificationActorProps, childName = "notification", minBackoff = 3.seconds, maxBackoff = 30.seconds, randomFactor = 0.2 // adds 20% "noise" to vary the intervals slightly ))
And then start singleton cluster like this:
system.actorOf( ClusterSingletonManager.props( singletonProps = supervisorProps, settings = ClusterSingletonManagerSettings(system)), name = "notification")
But I am not sure about some corner cases and pitfull of this approach.
So in general the question is - what is the proper approach to handle cluster singleton/ persistent actor supervision?