What is actually the prefered way to both sync up a new service A with data from another service B and also subscribe for further updates?
As I understand it, if A subscribes to B’s kafka topic it starts consuming from the “head” of the topic and does NOT consume messages from the “past”.
Therefore the only 2 solutions I can think of so far are these:
- Create a new topic just for A and let lagom fill the topic with the messages generated from the beginning of the event journal. For me it seems a bit ugly to always create a new topic for a new subscriber.
- Subscribe to the topic AND additionally fetch the current state of B (via HTTP API). This requires some ordering/sequence nr in kafka messages and HTTP response data to be able to correctly drop messages received on the kafka topic if they are “older” then the ones received in the current state response.
Both solutions have drawbacks.
I think thats a very common problem. Is there already a good solution for it?
I am hoping to find a way to replay a kafka topic only for given consumers. So my new service A will receive all messages generated from the beginning of the event journal but other consumers will “ignore” those “old” messages which they have already seen.