TopicProducer.taggedStreamWithOffset event published to Kafka do not have uniform time delay(lag)

(Amit) #1

We are publishing event to kafka using TopicProducer.taggedStreamWithOffset api. Out of 100 event , around 90 event is published to Kafka with lag of say(500-700ms) but rest 10 takes 2000 to 3000ms or even more.

Is there any setting which we can tune on Lagom Kafka client (producer) so that we experience uniform delay let say( 1000ms ) for all the event.

(Ignasi Marimon-Clos) #2

Hi @icxamit,

there is no setting since it’s not possible to guarantee a consistent delay. Lagom Read Side processors (and TopicProducer's) start a thread that polls the database. This polling strategy varies depending on the Akka Persistence Plugin (Cassandra, Couchbase, JDBC) implementation that you use.

In most cases (if not all), it’s possible that threads polling need to add a synthetic wait because the query finds a logical gap on the journal stream so a small wait is necessary while data sync’s across the nodes on the DB cluster.


Is there a setting or any way to control the frequency with which the TopicProducer thread polls for events in message table to publish to Kafka?