Akka-stream-kafka tuning for low latency

Akka
1

Hi,

We are using akka-stream-kafka lib for writing kafka subscribers and we have a low latency requirement i.e. below 10ms.

our kafka consumer subscribe API looks like this

private def subscribe(subscription: Subscription): Source[Event, scaladsl.Consumer.Control] =
    scaladsl.Consumer
      .plainSource(consumerSettings, subscription)
      .map(record ⇒ Event.fromPb(PbEvent.parseFrom(record.value())))

We are subscribing to around 200 topics from single JVM on which producer is producing events at varying rates for ex. 1 msg/sec, 100 msgs/sec.

I was looking at the akka-stream-kafka code and could see that for every subscription it creates KafkaConsumerActor which keeps polling to KafkaConsumer at configured poll interval. This means, in our case we are creating 200 streams which in turns create 200 actors which are continuously polling KafkaConsumer.

Does this adds overhead and cost latency, we are getting 99%tile latency around 200ms?

2

Hi,

you can try a couple of things to decrease the latency. You can resuse the consumer actor by creating the actor beforehand and then pass it in to source factory:

https://doc.akka.io/docs/akka-stream-kafka/current/consumer.html#sharing-the-kafkaconsumer-instance

Also you can tune the polling interval in the configuration:

1 Like
3

Thank you @2m for pointers.
By sharing ConsumerActor, 99%tile latency got reduced to 50ms which is also constant event if I increase number of publishers and subscribers to 250 publishing with rate of 1msg/sec.
But I think kafka is not made for such a low latency use cases (< 5ms) as it is poll based and broker needs to write logs to file, though this is a huge improvement from earlier results.