The copy an Akka Persistence Cassandra database doesn't work

Hi,

I have exported akka-persistance-cassandra database and loaded on another machine. When I send a new command with a new shard id or existing shard id, an entity doesn’t receiving the command. I did export/load using cassnadra COPY/FROM command. I can see it is hitting shard id extractor and entity id extractor.
Please, could anyone help me what could be the issue?

Thanks.

I got below error:

2709/09 09:27:59 ERROR[ngage-actor-system-akka.actor.default-dispatcher-22] a.c.s.PersistentShardCoordinator - Persistence failure when replaying events for persistenceId [/sharding/GroupEntityCoordinator]. Last known sequence number [0]
java.lang.NullPointerException: null
    at akka.persistence.journal.WriteJournalBase.adaptFromJournal(WriteJournalBase.scala:27)
    at akka.persistence.journal.WriteJournalBase.adaptFromJournal$(WriteJournalBase.scala:26)
    at akka.persistence.cassandra.journal.CassandraJournal.adaptFromJournal(CassandraJournal.scala:54)
    at akka.persistence.journal.AsyncWriteJournal$$anonfun$receiveWriteJournal$1.$anonfun$applyOrElse$15(AsyncWriteJournal.scala:147)
    at akka.persistence.journal.AsyncWriteJournal$$anonfun$receiveWriteJournal$1.$anonfun$applyOrElse$15$adapted(AsyncWriteJournal.scala:145)
    at akka.stream.impl.fusing.Map$$anon$1.onPush(Ops.scala:54)
    at akka.stream.impl.fusing.GraphInterpreter.processPush(GraphInterpreter.scala:523)
    at akka.stream.impl.fusing.GraphInterpreter.execute(GraphInterpreter.scala:409)
    at akka.stream.impl.fusing.GraphInterpreterShell.runBatch(ActorGraphInterpreter.scala:606)
    at akka.stream.impl.fusing.GraphInterpreterShell$AsyncInput.execute(ActorGraphInterpreter.scala:485)
    at akka.stream.impl.fusing.GraphInterpreterShell.processEvent(ActorGraphInterpreter.scala:581)
    at akka.stream.impl.fusing.ActorGraphInterpreter.akka$stream$impl$fusing$ActorGraphInterpreter$$processEvent(ActorGraphInterpreter.scala:749)
    at akka.stream.impl.fusing.ActorGraphInterpreter.akka$stream$impl$fusing$ActorGraphInterpreter$$shortCircuitBatch(ActorGraphInterpreter.scala:739)
    at akka.stream.impl.fusing.ActorGraphInterpreter$$anonfun$receive$1.applyOrElse(ActorGraphInterpreter.scala:765)
    at akka.actor.Actor.aroundReceive(Actor.scala:539)
    at akka.actor.Actor.aroundReceive$(Actor.scala:537)
    at akka.stream.impl.fusing.ActorGraphInterpreter.aroundReceive(ActorGraphInterpreter.scala:671)
    at akka.actor.ActorCell.receiveMessage(ActorCell.scala:612)
    at akka.actor.ActorCell.invoke(ActorCell.scala:581)
    at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:268)
    at akka.dispatch.Mailbox.run(Mailbox.scala:229)
    at akka.dispatch.Mailbox.exec(Mailbox.scala:241)
    at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
    at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
    at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
    at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)

Maybe you can verify that all data was included by running some cqlsh queries on the messages table.

select * from akka.messages where persistence_id='<a specific persistence id>' and partition_nr=1;

does it look the same as in the original database?

Yes, I can see the entry with same persistence id and partition_nr but for me, partition_nr is 0.

Ok, it probably starts at 0. All fields the same in old and new db?

Yeah. I checked entries are correct. Below are the debug logs.

12:30:00.163  DEBUG [ngage-actor-system-akka.actor.default-dispatcher-31] a.p.c.journal.CassandraJournal - [/sharding/UserFieldUniquenessCheckEntityCoordinator] asyncReplayMessages from [1] to [2]
12:30:00.165  DEBUG [ngage-actor-system-akka.actor.default-dispatcher-3] a.p.c.journal.CassandraJournal - [/sharding/UserFieldUniquenessCheckEntityCoordinator] Recovery sending tag progress: [Map()]
12:30:00.166  DEBUG [ngage-actor-system-akka.actor.default-dispatcher-3] a.p.cassandra.journal.TagWriters - Pid [/sharding/UserFieldUniquenessCheckEntityCoordinator] set tag progress [Map()]. Tags to reset as not in progress: [Set()]
12:30:00.166  DEBUG [ngage-actor-system-akka.actor.default-dispatcher-3] a.p.c.journal.CassandraJournal - [/sharding/UserFieldUniquenessCheckEntityCoordinator] Recovery is starting before the latest tag writes tag progress. Min progress [1]. From sequence nr of recovery: [1]
12:30:00.166  DEBUG [ngage-actor-system-akka.actor.default-dispatcher-3] a.p.c.journal.CassandraJournal - [/sharding/UserFieldUniquenessCheckEntityCoordinator] starting recovery with tag progress: [Map()]. From [1] to [2]
12:30:00.166  DEBUG [ngage-actor-system-akka.actor.default-dispatcher-31] a.p.c.q.s.CassandraReadJournal - Creating EventByPersistentIdState graph
12:30:00.169  DEBUG [ngage-actor-system-akka.actor.default-dispatcher-31] a.p.c.q.EventsByPersistenceIdStage - EventsByPersistenceId [/sharding/UserFieldUniquenessCheckEntityCoordinator] Query from seqNr [1] in partition [0]
12:30:00.171  DEBUG [ngage-actor-system-akka.actor.default-dispatcher-31] a.p.c.q.EventsByPersistenceIdStage - EventsByPersistenceId [/sharding/UserFieldUniquenessCheckEntityCoordinator] Query took [1] ms 
12:30:00.173  DEBUG [ngage-actor-system-akka.actor.default-dispatcher-28] a.p.cassandra.journal.TagWriters - Persistent actor terminated [Actor[akka://ngage-actor-system/system/sharding/UserFieldUniquenessCheckEntityCoordinator/singleton/coordinator#1767673975]]. Informing TagWriter actors to drop state for pid: [/sharding/UserFieldUniquenessCheckEntityCoordinator]
12:30:00.769  DEBUG [ngage-actor-system-akka.actor.default-dispatcher-28] a.p.c.journal.CassandraJournal - [/sharding/UserEntityCoordinator] asyncReadHighestSequenceNr [0] [Actor[akka://ngage-actor-system/system/sharding/UserEntityCoordinator/singleton/coordinator#-611062645]]
12:30:00.774  DEBUG [ngage-actor-system-akka.actor.default-dispatcher-28] a.p.c.journal.CassandraJournal - [/sharding/UserEntityCoordinator] Persistent actor starting [Actor[akka://ngage-actor-system/system/sharding/UserEntityCoordinator/singleton/coordinator#-611062645]]
12:30:00.774  DEBUG [ngage-actor-system-akka.actor.default-dispatcher-28] a.p.cassandra.journal.TagWriters - Watching pid [/sharding/UserEntityCoordinator] actor [Actor[akka://ngage-actor-system/system/sharding/UserEntityCoordinator/singleton/coordinator#-611062645]]
12:30:00.774  DEBUG [ngage-actor-system-akka.actor.default-dispatcher-28] a.p.c.journal.CassandraJournal - [/sharding/UserEntityCoordinator] asyncReplayMessages from [1] to [2]
12:30:00.777  DEBUG [ngage-actor-system-akka.actor.default-dispatcher-37] a.p.c.journal.CassandraJournal - [/sharding/UserEntityCoordinator] Recovery sending tag progress: [Map()]
12:30:00.777  DEBUG [ngage-actor-system-akka.actor.default-dispatcher-37] a.p.cassandra.journal.TagWriters - Pid [/sharding/UserEntityCoordinator] set tag progress [Map()]. Tags to reset as not in progress: [Set()]
12:30:00.778  DEBUG [ngage-actor-system-akka.actor.default-dispatcher-37] a.p.c.journal.CassandraJournal - [/sharding/UserEntityCoordinator] Recovery is starting before the latest tag writes tag progress. Min progress [1]. From sequence nr of recovery: [1]
12:30:00.778  DEBUG [ngage-actor-system-akka.actor.default-dispatcher-37] a.p.c.journal.CassandraJournal - [/sharding/UserEntityCoordinator] starting recovery with tag progress: [Map()]. From [1] to [2]
12:30:00.778  DEBUG [ngage-actor-system-akka.actor.default-dispatcher-37] a.p.c.q.s.CassandraReadJournal - Creating EventByPersistentIdState graph
12:30:00.781  DEBUG [ngage-actor-system-akka.actor.default-dispatcher-28] a.p.c.q.EventsByPersistenceIdStage - EventsByPersistenceId [/sharding/UserEntityCoordinator] Query from seqNr [1] in partition [0]
12:30:00.782  DEBUG [ngage-actor-system-akka.actor.default-dispatcher-28] a.p.c.q.EventsByPersistenceIdStage - EventsByPersistenceId [/sharding/UserEntityCoordinator] Query took [1] ms 
12:30:00.784  DEBUG [ngage-actor-system-akka.actor.default-dispatcher-3] a.p.cassandra.journal.TagWriters - Persistent actor terminated [Actor[akka://ngage-actor-system/system/sharding/UserEntityCoordinator/singleton/coordinator#-611062645]]. Informing TagWriter actors to drop state for pid: [/sharding/UserEntityCoordinator]
12:30:07.329  DEBUG [ngage-actor-system-akka.actor.default-dispatcher-31] a.p.c.journal.CassandraJournal - [/sharding/GroupEntityCoordinator] asyncReadHighestSequenceNr [0] [Actor[akka://ngage-actor-system/system/sharding/GroupEntityCoordinator/singleton/coordinator#-628561555]]
12:30:07.336  DEBUG [ngage-actor-system-akka.actor.default-dispatcher-31] a.p.c.journal.CassandraJournal - [/sharding/GroupEntityCoordinator] Persistent actor starting [Actor[akka://ngage-actor-system/system/sharding/GroupEntityCoordinator/singleton/coordinator#-628561555]]
12:30:07.336  DEBUG [ngage-actor-system-akka.actor.default-dispatcher-31] a.p.cassandra.journal.TagWriters - Watching pid [/sharding/GroupEntityCoordinator] actor [Actor[akka://ngage-actor-system/system/sharding/GroupEntityCoordinator/singleton/coordinator#-628561555]]
12:30:07.336  DEBUG [ngage-actor-system-akka.actor.default-dispatcher-31] a.p.c.journal.CassandraJournal - [/sharding/GroupEntityCoordinator] asyncReplayMessages from [1] to [2]
12:30:07.339  DEBUG [ngage-actor-system-akka.actor.default-dispatcher-20] a.p.c.journal.CassandraJournal - [/sharding/GroupEntityCoordinator] Recovery sending tag progress: [Map()]
12:30:07.339  DEBUG [ngage-actor-system-akka.actor.default-dispatcher-20] a.p.cassandra.journal.TagWriters - Pid [/sharding/GroupEntityCoordinator] set tag progress [Map()]. Tags to reset as not in progress: [Set()]
12:30:07.339  DEBUG [ngage-actor-system-akka.actor.default-dispatcher-20] a.p.c.journal.CassandraJournal - [/sharding/GroupEntityCoordinator] Recovery is starting before the latest tag writes tag progress. Min progress [1]. From sequence nr of recovery: [1]
12:30:07.339  DEBUG [ngage-actor-system-akka.actor.default-dispatcher-20] a.p.c.journal.CassandraJournal - [/sharding/GroupEntityCoordinator] starting recovery with tag progress: [Map()]. From [1] to [2]
12:30:07.339  DEBUG [ngage-actor-system-akka.actor.default-dispatcher-20] a.p.c.q.s.CassandraReadJournal - Creating EventByPersistentIdState graph
12:30:07.344  DEBUG [ngage-actor-system-akka.actor.default-dispatcher-20] a.p.c.q.EventsByPersistenceIdStage - EventsByPersistenceId [/sharding/GroupEntityCoordinator] Query from seqNr [1] in partition [0]
12:30:07.346  DEBUG [ngage-actor-system-akka.actor.default-dispatcher-20] a.p.c.q.EventsByPersistenceIdStage - EventsByPersistenceId [/sharding/GroupEntityCoordinator] Query took [2] ms 
12:30:07.348  DEBUG [ngage-actor-system-akka.actor.default-dispatcher-20] a.p.cassandra.journal.TagWriters - Persistent actor terminated [Actor[akka://ngage-actor-system/system/sharding/GroupEntityCoordinator/singleton/coordinator#-628561555]]. Informing TagWriter actors to drop state for pid: [/sharding/GroupEntityCoordinator]

Hi @patriknw do you think setting - cassandra-query-journal.gap-free-sequence-numbers = off
would help here? I saw you suggesting this in another thread where the issue was with RemoveInternalClusterShardingData

The reason I was asking for verifying that the transfer was correct was that the null pointer indicates that something is missing in the messages table.

gap-free-sequence-numbers is something else.

BTW, which version of akka-persistence-cassandra is this? Which version was used to write the data and which version is used now?

I am using the same akka-persistence-cassandra version 0.98 for while importing and exporting tables. Yes, transfer data is correct.
Let me explain bit more,
1- I have seed node that doesn’t host any entity and I have another node that host three entities. I also have another node, from there I am sending a commend to create an entity and I checked that entry has created successfully. I can see entries related to created entity.
2-I did stop everything including seed nodes.
3- I have script that exports all tables from akka and akka_snapshot keyspace.
4- I have removed previously created akka and akka_snapshot.
5- Loaded akka and akka_snapshots schema and data as well.
6- I started seed node, entity node and another node where there I was sending a commend to create entity
7- Now, I am sending command with same persistence id and got below error(I already posted same error)

12:56:32.875  DEBUG [ngage-actor-system-akka.actor.default-dispatcher-22] a.p.cassandra.journal.TagWriters - Pid [/sharding/GroupEntityCoordinator] set tag progress [Map()]. Tags to reset as not in progress: [Set()]
12:57:15.043  TRACE [ngage-actor-system-akka.actor.default-dispatcher-25] ngage.user.UserEntity - assigning state handler Init
12:57:15.044  TRACE [ngage-actor-system-akka.actor.default-dispatcher-25] ngage.user.UserEntity - assigning state handler Active
12:57:15.044  TRACE [ngage-actor-system-akka.actor.default-dispatcher-25] ngage.user.UserEntity - assigning state handler Inactive
12:57:15.045  TRACE [ngage-actor-system-akka.actor.default-dispatcher-25] ngage.user.UserEntity - assigning state handler Deleted
12:57:15.102  DEBUG [ngage-actor-system-akka.actor.default-dispatcher-25] a.p.c.journal.CassandraJournal - [User_0013] asyncReadHighestSequenceNr [0] [Actor[akka://ngage-actor-system/system/sharding/UserEntity/1/User_0013#151794441]]
12:57:15.108  DEBUG [ngage-actor-system-akka.actor.default-dispatcher-25] a.p.c.journal.CassandraJournal - [User_0013] Persistent actor starting [Actor[akka://ngage-actor-system/system/sharding/UserEntity/1/User_0013#151794441]]
12:57:15.109  DEBUG [ngage-actor-system-akka.actor.default-dispatcher-25] a.p.cassandra.journal.TagWriters - Watching pid [User_0013] actor [Actor[akka://ngage-actor-system/system/sharding/UserEntity/1/User_0013#151794441]]
12:57:15.110  DEBUG [ngage-actor-system-akka.actor.default-dispatcher-25] a.p.c.journal.CassandraJournal - [User_0013] asyncReplayMessages from [1] to [4]
12:57:15.135  DEBUG [ngage-actor-system-akka.actor.default-dispatcher-20] a.p.c.journal.CassandraJournal - [User_0013] Recovery sending tag progress: [Map()]
12:57:15.137  DEBUG [ngage-actor-system-akka.actor.default-dispatcher-20] a.p.cassandra.journal.TagWriters - Pid [User_0013] set tag progress [Map()]. Tags to reset as not in progress: [Set()]
12:57:15.142  DEBUG [ngage-actor-system-akka.actor.default-dispatcher-20] a.p.c.journal.CassandraJournal - [User_0013] Recovery is starting before the latest tag writes tag progress. Min progress [1]. From sequence nr of recovery: [1]
12:57:15.143  DEBUG [ngage-actor-system-akka.actor.default-dispatcher-20] a.p.c.journal.CassandraJournal - [User_0013] starting recovery with tag progress: [Map()]. From [1] to [4]
12:57:15.157  DEBUG [ngage-actor-system-akka.actor.default-dispatcher-25] a.p.c.q.s.CassandraReadJournal - Creating EventByPersistentIdState graph
12:57:15.189  DEBUG [ngage-actor-system-akka.actor.default-dispatcher-20] a.p.c.q.EventsByPersistenceIdStage - EventsByPersistenceId [User_0013] Query from seqNr [1] in partition [0]
12:57:15.192  DEBUG [ngage-actor-system-akka.actor.default-dispatcher-20] a.p.c.q.EventsByPersistenceIdStage - EventsByPersistenceId [User_0013] Query took [3] ms 
12:57:15.211  ERROR [ngage-actor-system-akka.actor.default-dispatcher-25] ngage.user.UserEntity - Persistence failure when replaying events for persistenceId [User_0013]. Last known sequence number [0]
java.lang.NullPointerException: null
	at akka.persistence.journal.WriteJournalBase.adaptFromJournal(WriteJournalBase.scala:27)
	at akka.persistence.journal.WriteJournalBase.adaptFromJournal$(WriteJournalBase.scala:26)
	at akka.persistence.cassandra.journal.CassandraJournal.adaptFromJournal(CassandraJournal.scala:54)
	at akka.persistence.journal.AsyncWriteJournal$$anonfun$receiveWriteJournal$1.$anonfun$applyOrElse$15(AsyncWriteJournal.scala:147)
	at akka.persistence.journal.AsyncWriteJournal$$anonfun$receiveWriteJournal$1.$anonfun$applyOrElse$15$adapted(AsyncWriteJournal.scala:145)
	at akka.stream.impl.fusing.Map$$anon$1.onPush(Ops.scala:54)
	at akka.stream.impl.fusing.GraphInterpreter.processPush(GraphInterpreter.scala:523)
	at akka.stream.impl.fusing.GraphInterpreter.execute(GraphInterpreter.scala:409)
	at akka.stream.impl.fusing.GraphInterpreterShell.runBatch(ActorGraphInterpreter.scala:606)
	at akka.stream.impl.fusing.GraphInterpreterShell$AsyncInput.execute(ActorGraphInterpreter.scala:485)
	at akka.stream.impl.fusing.GraphInterpreterShell.processEvent(ActorGraphInterpreter.scala:581)
	at akka.stream.impl.fusing.ActorGraphInterpreter.akka$stream$impl$fusing$ActorGraphInterpreter$$processEvent(ActorGraphInterpreter.scala:749)
	at akka.stream.impl.fusing.ActorGraphInterpreter.akka$stream$impl$fusing$ActorGraphInterpreter$$shortCircuitBatch(ActorGraphInterpreter.scala:739)
	at akka.stream.impl.fusing.ActorGraphInterpreter$$anonfun$receive$1.applyOrElse(ActorGraphInterpreter.scala:765)
	at akka.actor.Actor.aroundReceive(Actor.scala:539)
	at akka.actor.Actor.aroundReceive$(Actor.scala:537)
	at akka.stream.impl.fusing.ActorGraphInterpreter.aroundReceive(ActorGraphInterpreter.scala:671)
	at akka.actor.ActorCell.receiveMessage(ActorCell.scala:612)
	at akka.actor.ActorCell.invoke(ActorCell.scala:581)
	at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:268)
	at akka.dispatch.Mailbox.run(Mailbox.scala:229)
	at akka.dispatch.Mailbox.exec(Mailbox.scala:241)
	at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
	at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
	at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
	at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
12:57:15.220  DEBUG [ngage-actor-system-akka.actor.default-dispatcher-25] a.p.cassandra.journal.TagWriters - Persistent actor terminated [Actor[akka://ngage-actor-system/system/sharding/UserEntity/1/User_0013#151794441]]. Informing TagWriter actors to drop state for pid: [User_0013]
^_12:57:29.785  DEBUG [ngage-actor-system-akka.actor.default-dispatcher-4] a.p.cassandra.journal.TagWriters - Update tag scanning [(/sharding/UserFieldUniquenessCheckEntityCoordinator,1),(/sharding/UserEntityCoordinator,1),(/sharding/GroupEntityCoordinator,1)]
12:57:29.800  DEBUG [ngage-actor-system-akka.actor.default-dispatcher-4] a.p.cassandra.journal.TagWriters - Update tag scanning of [3] pids took [14] ms

For minimising the problem, I also try the below scenario:
1- I have a seed node that doesn’t host any entity and I have another node that host three entities. This time, I haven’t created any entity.
2-I did stop everything including seed nodes.
3- I have a script that exports all tables from akka and akka_snapshot keyspace.
4- I have removed previously created akka and akka_snapshot .
5- Loaded akka and akka_snapshots schema and data as well.
6- I started seed node, entity node and another node where there I was sending a command to create an entity
7- Now, I am sending a command and getting below the debug log and no entity has created, those logs seems to retry logs.

14:05:30.545  DEBUG [ngage-actor-system-akka.actor.default-dispatcher-24] a.p.c.journal.CassandraJournal - [/sharding/UserEntityCoordinator] Recovery is starting before the latest tag writes tag progress. Min progress [1]. From sequence nr of recovery: [1]
14:05:30.545  DEBUG [ngage-actor-system-akka.actor.default-dispatcher-24] a.p.c.journal.CassandraJournal - [/sharding/UserEntityCoordinator] starting recovery with tag progress: [Map()]. From [1] to [6]
14:05:30.546  DEBUG [ngage-actor-system-akka.actor.default-dispatcher-24] a.p.c.q.s.CassandraReadJournal - Creating EventByPersistentIdState graph
14:05:30.549  DEBUG [ngage-actor-system-akka.actor.default-dispatcher-32] a.p.c.q.EventsByPersistenceIdStage - EventsByPersistenceId [/sharding/UserEntityCoordinator] Query from seqNr [1] in partition [0]
14:05:30.550  DEBUG [ngage-actor-system-akka.actor.default-dispatcher-32] a.p.c.q.EventsByPersistenceIdStage - EventsByPersistenceId [/sharding/UserEntityCoordinator] Query took [2] ms 
14:05:30.554  DEBUG [ngage-actor-system-akka.actor.default-dispatcher-22] a.p.cassandra.journal.TagWriters - Persistent actor terminated [Actor[akka://ngage-actor-system/system/sharding/UserEntityCoordinator/singleton/coordinator#1872241626]]. Informing TagWriter actors to drop state for pid: [/sharding/UserEntityCoordinator]
14:05:36.759  DEBUG [ngage-actor-system-akka.actor.default-dispatcher-32] a.p.c.journal.CassandraJournal - [/sharding/GroupEntityCoordinator] asyncReadHighestSequenceNr [0] [Actor[akka://ngage-actor-system/system/sharding/GroupEntityCoordinator/singleton/coordinator#-1144391645]]
14:05:36.764  DEBUG [ngage-actor-system-akka.actor.default-dispatcher-32] a.p.c.journal.CassandraJournal - [/sharding/GroupEntityCoordinator] Persistent actor starting [Actor[akka://ngage-actor-system/system/sharding/GroupEntityCoordinator/singleton/coordinator#-1144391645]]
14:05:36.764  DEBUG [ngage-actor-system-akka.actor.default-dispatcher-32] a.p.cassandra.journal.TagWriters - Watching pid [/sharding/GroupEntityCoordinator] actor [Actor[akka://ngage-actor-system/system/sharding/GroupEntityCoordinator/singleton/coordinator#-1144391645]]
14:05:36.764  DEBUG [ngage-actor-system-akka.actor.default-dispatcher-32] a.p.c.journal.CassandraJournal - [/sharding/GroupEntityCoordinator] asyncReplayMessages from [1] to [4]
14:05:36.767  DEBUG [ngage-actor-system-akka.actor.default-dispatcher-24] a.p.c.journal.CassandraJournal - [/sharding/GroupEntityCoordinator] Recovery sending tag progress: [Map()]
14:05:36.767  DEBUG [ngage-actor-system-akka.actor.default-dispatcher-24] a.p.cassandra.journal.TagWriters - Pid [/sharding/GroupEntityCoordinator] set tag progress [Map()]. Tags to reset as not in progress: [Set()]
14:05:36.767  DEBUG [ngage-actor-system-akka.actor.default-dispatcher-24] a.p.c.journal.CassandraJournal - [/sharding/GroupEntityCoordinator] Recovery is starting before the latest tag writes tag progress. Min progress [1]. From sequence nr of recovery: [1]
14:05:36.768  DEBUG [ngage-actor-system-akka.actor.default-dispatcher-24] a.p.c.journal.CassandraJournal - [/sharding/GroupEntityCoordinator] starting recovery with tag progress: [Map()]. From [1] to [4]
14:05:36.768  DEBUG [ngage-actor-system-akka.actor.default-dispatcher-24] a.p.c.q.s.CassandraReadJournal - Creating EventByPersistentIdState graph
14:05:36.770  DEBUG [ngage-actor-system-akka.actor.default-dispatcher-24] a.p.c.q.EventsByPersistenceIdStage - EventsByPersistenceId [/sharding/GroupEntityCoordinator] Query from seqNr [1] in partition [0]
14:05:36.772  DEBUG [ngage-actor-system-akka.actor.default-dispatcher-24] a.p.c.q.EventsByPersistenceIdStage - EventsByPersistenceId [/sharding/GroupEntityCoordinator] Query took [1] ms 
14:05:36.775  DEBUG [ngage-actor-system-akka.actor.default-dispatcher-34] a.p.cassandra.journal.TagWriters - Persistent actor terminated [Actor[akka://ngage-actor-system/system/sharding/GroupEntityCoordinator/singleton/coordinator#-1144391645]]. Informing TagWriter actors to drop state for pid: [/sharding/GroupEntityCoordinator]

Hi @patriknw ,
I have created a simple project by copying akka-sample-persistence-scala from https://github.com/akka/akka-samples/tree/2.5/akka-sample-persistence-scala to produce similar issue. Here is a sample project https://github.com/abdheshkumar/akka-sample-persistence.
Steps to run:
1- src/main/resources/docker-compose up
2- Run PersistentActorExample.scala
3- src/main/resources/cassandra-import.sh //This will import tables on local machine from docker
4- src/main/resources/docker-compose down //For cleaning up
5 - src/main/resources/load.sh //Load cassandra tables into cassnadra
6- Re-run PersistentActorExample.scala
7. Will see an error