Akka Cassandra Persistence WriteTimeoutException in query with consistency SERIAL during single-node akka cluster startup

(Ivar Suisalu) #1

During single node cluster startup application fails with:

[error] a.c.s.PersistentShardCoordinator - Persistence failure when replaying events for persistenceId [/sharding/kafkaProducer-rawParameterValuesCoordinator]. Last known sequence number [0]
java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.WriteTimeoutException: Cassandra timeout during write query at consistency SERIAL (1 replica were required but only 0 acknowledged the write)

Cassandra instance is running in docker container and network connectivity to it seems to be ok. I wonder from where this SERIAL consistency comes from? In my mind this can be set on client side, but i didn’t find any relevant code in akka-persistence-cassandra source. And what could be the reason for this exception.
Cassandra version is 3.11, akka-persistence-cassandra i tryed 0.60 and 0.80.

So far i found that exception happens with CQL: “insert into item.config(property, value) values(?,?) if not exists”. The item is the persistent enty name. I din’t find the place in akka code where this cql is generated and what to do with it. One possibility is to change cassandra config prop write_query_timeout or something else. Question remains why in dev mode with embedded cassandra i don’t receive this exception.

(Patrik Nordwall) #2

It stores the partition size in that table at startup using IF NOT EXISTS and then I think all replicas must be available. We have considered removing this “feature”. I think it’s only supposed to do this the very first time. At next startup it should detect that the value has been written and then don’t try again (unless you recreate the keyspace).