Is it possible to share KafkaConsumerActor in Consumer.committableSource?

Akka Akka Streams & Alpakka
, ,
1

I am building an application where I have 100+ streams, one stream corresponding to each kafka topic. I would like to control each stream independently, so individual streams are better option. If KafkaConsumer is not shared, each stream creates a new Consumer object which internally creates one heartbeat thread per stream. This means that the application would have 100+ heartbeat threads.

I tried using the Consumer.committableExternalSource where we can share the kafkaConsumerActor, but that causes a separate set of challenges. With this, the subscription becomes manual, so offset tracking and assignment becomes the application’s responsibility. Also, this api doesn’t support rebalancing which again becomes the application’s responsibility.

Is there any reason why KafkaConsumerActor is not sharable in Consumer.committableSource? Is there a workaround to share the kafkaConsumerActor in committableSource? I looked at the alpakka-kafka source, but I couldn’t find a way to override this behaviour.

2

It is not possible right now, but should be easy to add. The actor creation can be moved out/made optional in SingleSourceLogic and where needed an actor ref could be passed in instead of creating it.

Open an issue in Alpakka Kafka and suggest the necessary changes in a pull-request.

– Enno.
Alpakka Team

3

Thank you Enno. I have created a Issue a few days ago. Here is the link: https://github.com/akka/alpakka-kafka/issues/707.
There are no updates so far. Can you please review and let me know if it needs more information ?

If I can provide a fix, I will open a pull request. Meanwhile, if any other developer picks it up, the issue description should help the developer in making the necessary changes.