Clearing data (Persistence failure when replaying events for persistenceId [MyEntityENTITY_KEY]. Last known sequence number [0])

Lagom
1

Hi,

I’ve tried clearing our staging server’s data and when I started our microservices, I keep getting this. Any ideas what Im doing wrong in terms of clearing our database?

Persistence failure when replaying events for persistenceId [MyEntityENTITY_KEY]. Last known sequence number [0]

(Note: our staging servers are in ubuntu)

How I clear our data

  1. Stop all services
    1.1. sudo systemctl stop zookeeper
    1.2. sudo systemctl stop kafka
    1.3. sudo systemctl stop cassandra
  2. Clear all data
    2.1. sudo rm -rf /var/lib/kafka/data/*
    2.2. sudo find /var/lib/zookeeper -type f -exec rm {} ';
    2.3. rm -rf /var/lib/cassandra/data/; rm -rf /var/lib/cassandra/commitlog/; find /var/lib/cassandra/ -type f -exec rm {} ‘;’
  3. Start all services
    3.1. sudo systemctl start zookeeper
    3.2. sudo systemctl start kafka
    3.3. sudo systemctl start cassandra
2

@franz do you also stop/start lagom services while doing this?

3

Yes. When I did this, all lagom services are down.

4

@franz can you share complete error stacktrace?
Also are you using:
akka.cluster.sharding.state-store-mode = ddata
?

5

Yes. We have akka.cluster.sharding.state-store-mode = ddata in our confs.

Here’s a sample stack trace

2018-10-29 14:24:43,837 ERROR c.l.l.i.j.p.PersistentEntityActor Persistence failure when replaying events for persistenceId MyEntityENTITY_KEY]. Last known sequence number [0]
akka.pattern.CircuitBreaker$$anon$1: Circuit Breaker Timed out.
2018-10-29 14:24:47,196 ERROR a.a.OneForOneStrategy Ask timed out on [Actor[akka://application/user/cassandraOffsetStorePrepare-singletonProxy#-671150673]] after [20000 ms]. Sender[null] sent message of type "com.lightbend.lagom.internal.persistence.cluster.ClusterStartupTaskActor$Execute$".
akka.pattern.AskTimeoutException: Ask timed out on [Actor[akka://application/user/cassandraOffsetStorePrepare-singletonProxy#-671150673]] after [20000 ms]. Sender[null] sent message of type "com.lightbend.lagom.internal.persistence.cluster.ClusterStartupTaskActor$Execute$".
	at akka.pattern.PromiseActorRef$.$anonfun$defaultOnTimeout$1(AskSupport.scala:596)
	at akka.pattern.PromiseActorRef$.$anonfun$apply$1(AskSupport.scala:606)
	at akka.actor.Scheduler$$anon$4.run(Scheduler.scala:205)
	at scala.concurrent.Future$InternalCallbackExecutor$.unbatchedExecute(Future.scala:866)
	at scala.concurrent.BatchingExecutor.execute(BatchingExecutor.scala:109)
	at scala.concurrent.BatchingExecutor.execute$(BatchingExecutor.scala:103)
	at scala.concurrent.Future$InternalCallbackExecutor$.execute(Future.scala:864)
	at akka.actor.LightArrayRevolverScheduler$TaskHolder.executeTask(LightArrayRevolverScheduler.scala:328)
	at akka.actor.LightArrayRevolverScheduler$$anon$4.executeBucket$1(LightArrayRevolverScheduler.scala:279)
	at akka.actor.LightArrayRevolverScheduler$$anon$4.nextTick(LightArrayRevolverScheduler.scala:283)
	at akka.actor.LightArrayRevolverScheduler$$anon$4.run(LightArrayRevolverScheduler.scala:235)
	at java.lang.Thread.run(Thread.java:748)
2018-10-29 14:24:47,233 ERROR a.a.OneForOneStrategy Ask timed out on [Actor[akka://application/user/cassandraOffsetStorePrepare-singletonProxy#-671150673]] after [20000 ms]. Sender[null] sent message of type "com.lightbend.lagom.internal.persistence.cluster.ClusterStartupTaskActor$Execute$".
akka.pattern.AskTimeoutException: Ask timed out on [Actor[akka://application/user/cassandraOffsetStorePrepare-singletonProxy#-671150673]] after [20000 ms]. Sender[null] sent message of type "com.lightbend.lagom.internal.persistence.cluster.ClusterStartupTaskActor$Execute$".
	at akka.pattern.PromiseActorRef$.$anonfun$defaultOnTimeout$1(AskSupport.scala:596)
	at akka.pattern.PromiseActorRef$.$anonfun$apply$1(AskSupport.scala:606)
	at akka.actor.Scheduler$$anon$4.run(Scheduler.scala:205)
	at scala.concurrent.Future$InternalCallbackExecutor$.unbatchedExecute(Future.scala:866)
	at scala.concurrent.BatchingExecutor.execute(BatchingExecutor.scala:109)
	at scala.concurrent.BatchingExecutor.execute$(BatchingExecutor.scala:103)
	at scala.concurrent.Future$InternalCallbackExecutor$.execute(Future.scala:864)
	at akka.actor.LightArrayRevolverScheduler$TaskHolder.executeTask(LightArrayRevolverScheduler.scala:328)
	at akka.actor.LightArrayRevolverScheduler$$anon$4.executeBucket$1(LightArrayRevolverScheduler.scala:279)
	at akka.actor.LightArrayRevolverScheduler$$anon$4.nextTick(LightArrayRevolverScheduler.scala:283)
	at akka.actor.LightArrayRevolverScheduler$$anon$4.run(LightArrayRevolverScheduler.scala:235)
	at java.lang.Thread.run(Thread.java:748)
6

@franz / @aklikic , Did you find a solution for this?.
I am also running into the same issue.

7

@franz @MalluGolageri ,Did you manage to find a solution for this. I have the same problem currently.

8

@Melissa, No. I did not find a solution for this yet.