Dev Mode Kafka Data Retention

Lagom Lagom Message Broker API
1

Hi All,

I’ve got a bit of an unusual circumstance and one of my services is talking directly to Kafka.

I’ve been trying to monitor the kafka topic in dev mode using the “kafka-console-consumer” provided by kafka, explicitly setting it to the only partition & topic I’ve allowed and telling it to play from offset “earliest”.

kafka-console-consumer.sh --topic=machine --bootstrap-server localhost:9092 --offset earliest --partition 0

For a brief window of about a minute, it successfully pulls down the messages, even several times, but after about a minute of running (after allowing the event-generating process to finish), re-running produces nothing, all of the old messages are gone.

Its my understanding that lagom’s dev-mode Kafka is supposed to be configured by default to retain messages for a while, the log.retention.hours=168 line in the settings I thought meant it should last for a week.

I realize this may be more of a kafka problem than a lagom problem, but as I’m also worried about what it might do to other services relying on replaying those messages, and it’s lagom’s dev-mode configuration, I figured I’d ask here first, in case my basic assumptions were flawed.

Thanks!

2

(I’m way out of my comfort zone here)

Maybe restarting kafka-console-consumer.sh produces nothing because the same consumer ID is used and the broker already knows that existing messages were already consumed (and offsets committed?). IIRC, earliest is used for new consumer IDs.

Please take this answer with a huge grain of salt.

ign

3

I think @ignasi35 is right. You can pass a --delete-consumer-offsets option to kafka-console-consumer.sh to clear the committed offset. Does that help?

4

I’ve had to put this task aside for the moment, but will be returning to it next week. I’ll give it a try. Thanks!