TagWriter fails if cassandra isn't started before lagom service

Akka Persistence / Event Sourcing
1

I’m seeing an issue where tags aren’t written if C* isn’t ready before starting the service. The TagWriter fails with:

hello_1      | 2018-05-03T03:16:28.977Z [warn] akka.persistence.cassandra.journal.TagWriter [sourceThread=application-
lagom.persistence.dispatcher-43, akkaTimestamp=03:16:28.976UTC, akkaSource=akka.tcp://application@172.26.0.20:2552/system
/cassandra-journal/tagWrites/com.example.hello.impl.HelloEvent, sourceActorSystem=application] - Writing tags has failed. 
This means that any eventsByTag query will be out of date. The write will be retried. Reason 
com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: /172.26.0.10:9042
(com.datastax.driver.core.exceptions.TransportException: [/172.26.0.10:9042] Cannot connect))

What’s odd is that writing events to the journal works - I’m not sure why it says no hosts are available. If C* is up and ready when starting the service then it works. If you restart the failing service then it will work.

It’s all in a docker-compose setup and I have a reproduction at https://github.com/jibbers42/lagom-write-tag-fail. I’m using:

  • lagom 1.4.4
  • akka-persistence-cassandra 0.84 - it may be worth noting that I forgot to update to 0.84 when creating the repro and couldn’t reproduce the error, but I’m not sure if whatever version of akka-persistence-cassandra that lagom comes with even uses TagWriter.

See https://github.com/jibbers42/lagom-write-tag-fail/blob/master/README.md for more details and steps to reproduce.

Any ideas what might be causing this?

2

Hi @jibbers42,

You are correct that TagWriter is a new change in Akka Persistence Cassandra. Lagom has not yet been fully tested with this version of Akka Persistence Cassandra, so there might be some rough edges. I’m going to move this topic to the Akka Persistence category to make it more visible to the Akka Persistence Cassandra experts.