My Bug while Using Multiple Cluster sharding with Akka Persistence

Hello, dear Akka gurus.
I’m dealing with a bad bug that I hesitate if it is in my code or not.
Consider I have three types of Entities (Actors :slight_smile:).

  • Type A: Akka’s Default ClusterSharding
  • Type B & C: KafkaClusterSharding

and all of them are persistent actors using the Cassandra persistence plugin. I see strange behavior in my tests:

  1. actor b from type B: spawn and respond to some messages
  2. perhaps actor b stops.
  3. actor b: recoveries with State of type A and could not replay the events and I face this error log (Disclaimer: I don’t use any EventAdapter):

[akka://myssys/system/sharding/A/1/c381b116-ba2a-403f-86d2-ef73d0c83e9a] - Initializing snapshot recovery: Recovery(SnapshotSelectionCriteria(9223372036854775807,9223372036854775807,0,0),9223372036854775807,9223372036854775807)
[akka://myssys/system/sharding/A/1/c381b116-ba2a-403f-86d2-ef73d0c83e9a] - Replaying events: from: 1, to: 9223372036854775807
[akka://myssys/system/sharding/A/1/c381b116-ba2a-403f-86d2-ef73d0c83e9a#111024197]]. Persistent Actors with the same PersistenceId should not run concurrently
[akka://myssys/system/sharding/A/1/c381b116-ba2a-403f-86d2-ef73d0c83e9a] - Returning recovery permit, reason: on replay failure: No match found for event [class B$B_Event_1] and state [B$State]. Has this event been stored using an EventAdapter? (of class java.lang.String)
[akka://myssys/system/sharding/A/1/c381b116-ba2a-403f-86d2-ef73d0c83e9a] - Recovery failure for persistenceId [PersistenceId(c381b116-ba2a-403f-86d2-ef73d0c83e9a)] after 59.05 ms
[akka://myssys/system/sharding/A/1/c381b116-ba2a-403f-86d2-ef73d0c83e9a] - Recovery failure for persistenceId [PersistenceId(c381b116-ba2a-403f-86d2-ef73d0c83e9a)] after 60.18 ms
[akka://myssys/system/sharding/A/1/c381b116-ba2a-403f-86d2-ef73d0c83e9a] - Supervisor RestartSupervisor saw failure: Exception during recovery. Last known sequence number [1]. PersistenceId [c381b116-ba2a-403f-86d2-ef73d0c83e9a], due to: Exception during recovery while handling [B$B_Event_1] with sequence number [1]. PersistenceId [c381b116-ba2a-403f-86d2-ef73d0c83e9a], due to: No match found for event [class B$B_Event_1] and state [A$State]. Has this event been stored using an EventAdapter? (of class java.lang.String)
akka.persistence.typed.internal.JournalFailureException: Exception during recovery. Last known sequence number [1]. PersistenceId [c381b116-ba2a-403f-86d2-ef73d0c83e9a], due to: Exception during recovery while handling [B$B_Event_1] with sequence number [1]. PersistenceId [c381b116-ba2a-403f-86d2-ef73d0c83e9a], due to: No match found for event [class B$B_Event_1] and state [A$State]. Has this event been stored using an EventAdapter? (of class java.lang.String)
at akka.persistence.typed.internal.ReplayingEvents.onRecoveryFailure(ReplayingEvents.scala:220)
at akka.persistence.typed.internal.ReplayingEvents.onJournalResponse(ReplayingEvents.scala:153)

What is my Problem? Please help me.