Persistence failure when replaying events and Failed to persist event with jdbc

Akka Persistence / Event Sourcing
,
1

Hi

On a raspberry pi compute module 3, we run akka classic with persistence (please find versions below).
Persistence is very unreliable and we don’t know why. The errors we get are of the following two types:

  1. Persistence failure when replaying events for persistenceId [someId}]. Last known sequence number [7000] akka.pattern.CircuitBreaker$$anon$13: Circuit Breaker Timed out.
  2. Failed to persist event type [someEvent] with sequence number [76817] for persistenceId [someId] akka.pattern.CircuitBreaker$$anon$13: Circuit Breaker Timed out.

Our slick configuration in application.conf looks like:

slick {
  profile = "slick.jdbc.PostgresProfile$"
  db {
    url = "jdbc:postgresql://localhost:5432/databaseName?reWriteBatchedInserts=true"
    user = "someUser"
    password = "somePW"
    driver = "org.postgresql.Driver"

    // hikariCP
    numThreads = 4 // number of cores
    maxConnections = 4 // limit maxConnections to numThreads to avoid deadlocks
  }
}

Facts

  1. there are only 7 persistent actors
  2. we use postgresql as data storage, installed on the raspberry pi compute module
  3. the errors occur with both versions of postgresql: 9.4 and 11.11
  4. every minute, a cronjob removes entries marked as deleted from the postgresql table event_journal to prevent full storage
  5. we apply the standard schema to our postgresql database

Versions

akka version 2.6.10
scala version 2.13
slick version 3.3.3
akka-persistence-jdbc version 5.0.0
postgresql version 42.2.14 (from org.postgresql using maven)

Questions

  1. How can we get more insight into what might cause the problem? (like enable more persistence logs, slick logs, …)
  2. What can you tell from the error logs?
  3. is there some configuration options to solve the problems?

Thanks