Cassandra Journal: eventsByTag receiving events 5 min after startup

Akka version: 2.6.4
Akka Persistence Cassandra version: 1.0.0-RC1

We are leveraging CassandraReadJournal’s eventsByTag to stream the events to a message broker. The problem we are facing is that after starting the service and emitting the first event from the EventSourcedBehavior, eventsByTag does not receive any events after 5 minutes. Subsequent events published afterwards are almost immediately received though.

This is the log message that indicates eventsByTag was subscribed to:

{"@timestamp":“2020-04-28T08:01:19.294Z”,
“message”:"[e133518f-0599-4426-9d15-f3a6054904a5]: EventsByTag query [Onboarding]
starting with EC delay 5000ms: fromOffset [a4698000-8f19-11e5-8080-808080808080 (2015-11-20 00:00:00:000)] toOffset [None]",“logger_name”:“a.p.cassandra.query.EventsByTagStage”,“thread_name”:“OnboardingService-akka.actor.default-dispatcher-14”,“level”:“INFO”,“akkaAddress”:“akka://OnboardingService@172.17.0.15:25520”,“sourceThread”:“OnboardingService-akka.actor.default-dispatcher-21”,“akkaSource”:“EventsByTagStage(akka://OnboardingService)”,“sourceActorSystem”:“OnboardingService”,“akkaTimestamp”:“08:01:19.294UTC”}

This log messages is appended by the EventSourcedBehavior on event emitted:

{"@timestamp":“2020-04-28T08:02:58.117Z”,
“message”:“Actor emitted event OnboardingCreated{id=52f5da2d-ed8f-4f3a-a9b2-6c6f452ac363, idempotencyKey=‘9a6f08df-4810-4f58-838d-1eb8bfe10cff’}”,“logger_name”:“r.s.o.a.c.p.OnboardIdentityRequestProcessor”,“thread_name”:“OnboardingService-akka.actor.default-dispatcher-14”,“level”:“INFO”}

And around 4 minutes after the event above, it is received by the event subscriptor via eventsByTag:

{"@timestamp":“2020-04-28T08:06:05.277Z”,
“message”:“Successfully published message 391:2:-1:0”,“logger_name”:“r.s.o.actor.query.EventProcessor”,“thread_name”:“OnboardingService-akka.actor.default-dispatcher-12”,“level”:“INFO”}

I haven’t applied any custom configuration to the Cassandra journal. Is that 4-5 min initial delay expected? Is there a way this can be configured. I need to mention we don’t experience this issue when using the in-memory journal.

1 Like

That’s definitely not expected!

If you turn on DEBUG logging you should see the events by tag stage constantly doing queries to look for events, do you see those?