We are using Akka Persistence with Cassandra 3.x. During application (pod in Openshift) startup, we have noticed occasional Circuit Breaker timeout issues. Few questions
-
When we have multiple Cassandra contact points, and circuit breaker condition is met for one node – will recovery resume or restart using another Cassandra node?
-
When recovery fails, is there a way to get Cassandra connection information? I mean, recovery failed using which connection.
-
If there is no self-healing, is there a way for the application to restart recovery and force another Cassandra node?
Library Details
- akka-persistence_2.12: 2.5.12
- akka-persistence-cassandra_2.12: 0.85
Circuit Breaker Configuration
circuit-breaker {
max-failures = 5
call-timeout = 1s
reset-timeout = 1s
}
Error Logs
2018-11-06T14:59:26,455 ERROR [csm-akka.actor.default-dispatcher-4](akka://csm/user/network) -- system.NetworkSupervisor - Persistence failure when replaying events for persistenceId [network]. Last known sequence number [196]
akka.pattern.CircuitBreaker$$anon$1: Circuit Breaker Timed out.
2018-11-06T14:59:26,466 ERROR [csm-akka.actor.default-dispatcher-24](akka://csm/user/network) -- actor.OneForOneStrategy - Recovery Failed: for the Event::None::Circuit Breaker Timed out.
csm.exceptions.RecoveryFailedException: Recovery Failed: for the Event::None::Circuit Breaker Timed out.
at csm.actors.system.AbstractClientStateSupervisingActor.onRecoveryFailure(AbstractClientStateSupervisingActor.java:56) ~[main/:?]
at akka.persistence.Eventsourced$$anon$4.stateReceive(Eventsourced.scala:623) ~[akka-persistence_2.12-2.5.13.jar:2.5.13]
at akka.persistence.Eventsourced.aroundReceive(Eventsourced.scala:222) ~[akka-persistence_2.12-2.5.13.jar:2.5.13]
at akka.persistence.Eventsourced.aroundReceive$(Eventsourced.scala:221) ~[akka-persistence_2.12-2.5.13.jar:2.5.13]
at akka.persistence.AbstractPersistentActor.aroundReceive(PersistentActor.scala:454) ~[akka-persistence_2.12-2.5.13.jar:2.5.13]
at csm.actors.system.AbstractClientStateSupervisingActor.aroundReceive(AbstractClientStateSupervisingActor.java:49) ~[main/:?]
at akka.actor.ActorCell.receiveMessage(ActorCell.scala:588) [akka-actor_2.12-2.5.13.jar:2.5.13]
at akka.actor.ActorCell.invoke(ActorCell.scala:557) [akka-actor_2.12-2.5.13.jar:2.5.13]
at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:258) [akka-actor_2.12-2.5.13.jar:2.5.13]
at akka.dispatch.Mailbox.run(Mailbox.scala:225) [akka-actor_2.12-2.5.13.jar:2.5.13]
at akka.dispatch.Mailbox.exec(Mailbox.scala:235) [akka-actor_2.12-2.5.13.jar:2.5.13]
at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) [akka-actor_2.12-2.5.13.jar:2.5.13]
at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) [akka-actor_2.12-2.5.13.jar:2.5.13]
at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) [akka-actor_2.12-2.5.13.jar:2.5.13]
at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) [akka-actor_2.12-2.5.13.jar:2.5.13]