Public API Consumer consumes twice after restart


(Michael Mangeng) #1

Hi all!

I’ve got a problem with a occasional repetition of events by an lagom topic consumer after the restart of the service.

I’ve got a “UserService” which emits an “UserActivationEvent”. This event is published on a topic which is subscribed by a Consumer (in the same package as the UserService). This consumer sends an Activation-Mail for every UserActivationEvent.

If i restart this service i occasionally get new mails which have already been sent some time ago. Just now i checked the offset of this consumer in kafka (which was 1) and restarted the service. Now i got a second activation mail for this event which should not be the case because the only event in the topic is also offset 1.
Currently i’m not able to reproduce it. If i restart it several times, it does not happen. Looks like some time needs to pass…

The kafka topic retention time is 7 days. I created the event on 20. Nov (today is 26.) - so this cannot be an issue. And Kafka has not been restarted.

The code of my consumer is: https://pastebin.com/cK026PUe

Would be very glad if somebody could drop me a hint what the problem could be or where i can do some debugging because this is a major showstopper for us.

greetings,
Michael

PS: Lagom Kafka Broker settings are:

lagom.broker.kafka {
service-name = “”

brokers = “10.41.23.10:9092”

client {
default {
failure-exponential-backoff {
min = 3s
max = 30s
random-factor = 0.2
}
}

producer = ${lagom.broker.kafka.client.default}
producer.role = ""

consumer {
  failure-exponential-backoff = ${lagom.broker.kafka.client.default.failure-exponential-backoff}
  offset-buffer = 100
  batching-size = 20
  batching-interval = 5 seconds
}

}
}


(Alan Klikic) #2

@mm
Check kafka consumer offset retention. I had problems when kafka topic retention was longer then offset retention. So i have increased offset retation to max retention of all topics.

Consumer offset is related to consumer group (in your case auth-ActivationEventConsumer).

If consumer group did not change (you did not change withGroupId value) and offset looks ok for this consumer group on kafka then I would try consuming the topic with kafka console client (using same consumer group) to check how it is consumed there.