12:13:44.593 [warn] akka.persistence.cassandra.journal.CassandraJournal [sourceThread=hello-impl-application-lagom.persistence.dispatcher-29, akkaTimestamp=11:13:44.592UTC, akkaSource=akka.tcp://hello-impl-application@127.0.0.1:63213/system/cassandra-journal, sourceActorSystem=hello-impl-application] - Failed to connect to Cassandra and initialize. It will be retried on demand. Caused by: com.datastax.driver.core.exceptions.WriteTimeoutException: Cassandra timeout during CAS write query at consistency SERIAL (1 replica were required but only 0 acknowledged the write)
Trying to call any REST Api, throws:
akka.pattern.AskTimeoutException: Ask timed out on [Actor[akka://hello-impl-application/system/sharding/HelloEntity#-1282607955]] after [5000 ms]. Message of type [com.lightbend.lagom.scaladsl.persistence.CommandEnvelope]. A typical reason for AskTimeoutException is that the recipient actor didn’t send a reply.
at akka.pattern.PromiseActorRef$.$anonfun$defaultOnTimeout$1(AskSupport.scala:648)
at akka.pattern.PromiseActorRef$.$anonfun$apply$1(AskSupport.scala:669)
at akka.actor.Scheduler$$anon$4.run(Scheduler.scala:202)
at scala.concurrent.Future$InternalCallbackExecutor$.unbatchedExecute(Future.scala:874)
at scala.concurrent.BatchingExecutor.execute(BatchingExecutor.scala:113)
at scala.concurrent.BatchingExecutor.execute$(BatchingExecutor.scala:107)
at scala.concurrent.Future$InternalCallbackExecutor$.execute(Future.scala:872)
at akka.actor.LightArrayRevolverScheduler$TaskHolder.executeTask(LightArrayRevolverScheduler.scala:334)
at akka.actor.LightArrayRevolverScheduler$$anon$3.executeBucket$1(LightArrayRevolverScheduler.scala:285)
at akka.actor.LightArrayRevolverScheduler$$anon$3.nextTick(LightArrayRevolverScheduler.scala:289)
at akka.actor.LightArrayRevolverScheduler$$anon$3.run(LightArrayRevolverScheduler.scala:241)
at java.lang.Thread.run(Thread.java:748)
Having it run with a Cassandra database deployed on the localhost doesn’t go into any timeout… Any clue or documentation would be much appreciated as we would like to see how far lagom can be deployed on a local kubernetes cluster including cassandra within the cluster.
Also the weirdest thing about it, is that the app keyspace and related cqrs tables are created while going through cqlsh on the cassandra instance deployed on the minikube cluster.
This is during initialization, are any queries succeeding? Often when you get a timeout with replicas available 9 it is because your keyspace referes to data centers that don’t exist in the cluster.
Describe the keyspace and check that if it is using NetworkTopology that the datacenter than nodetool reports with nodetool status
Most of the initialisation is succeeding as I can see that Lagom creates the app keyspace and related tables. Could you please illustrate, ideally with a command, which information should be checked with nodetool ?
Meanwhile, Here after the ouput of a cqlsh to the cassandra instance on minikube, my app keyspace being hello:
Connected to cassandra at 127.0.0.1:9042.
[cqlsh 5.0.1 | Cassandra 3.11.5 | CQL spec 3.4.4 | Native protocol v4]
Use HELP for help.
cassandra@cqlsh> describe keyspaces
stream system_auth system_traces **hello**
system_schema system system_distributed authtest
Still getting a timeout error to the Akka Actor (PersistentEntity), when calling the hello rest api:
13:50:14.375 [error] akka.cluster.sharding.PersistentShardCoordinator [sourceThread=poste-impl-application-akka.actor.default-dispatcher-17, akkaTimestamp=12:50:14.373UTC, akkaSource=akka.tcp://poste-impl-application@127.0.0.1:62618/system/sharding/PosteEntityCoordinator/singleton/coordinator, sourceActorSystem=poste-impl-application] - Persistence failure when replaying events for persistenceId [/sharding/PosteEntityCoordinator]. Last known sequence number [0]
java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.OperationTimedOutException: [/127.0.0.1:9042] Timed out waiting for server response
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:552)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:513)
at akka.persistence.cassandra.package$$anon$1.$anonfun$run$1(package.scala:18)
at scala.util.Try$.apply(Try.scala:213)
at akka.persistence.cassandra.package$$anon$1.run(package.scala:18)
at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:41)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: com.datastax.driver.core.exceptions.OperationTimedOutException: [/127.0.0.1:9042] Timed out waiting for server response
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onTimeout(RequestHandler.java:954)
at com.datastax.driver.core.Connection$ResponseHandler$1.run(Connection.java:1575)
at io.netty.util.HashedWheelTimer$HashedWheelTimeout.expire(HashedWheelTimer.java:682)
at io.netty.util.HashedWheelTimer$HashedWheelBucket.expireTimeouts(HashedWheelTimer.java:757)
at io.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:485)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
The weirdest thing about it, is that running the Hello Project (same sample of code) using the Java seed project from Lagom (https://www.lagomframework.com/get-started-java-maven.html), just works, unlike the scala implementation with sbt…
