I am running a simulation which generates a million records every second. I’m writing them to Kafka and reading them through Akka Streams. I’m performing a few aggregations on this data and writing the output back to Kafka.
The data contains a timestamp based on which the aggregations are grouped. Using the timestamps, I’m creating windows of data and performing aggregation on these windows. Since there are a million records each second, the aggregations are taking about 40 seconds for one million records. This is really slow because new data is being generated and written to Kafka every second.
I referred this blog post for performing window aggregations.
Is there any better way to perform these aggregations in lesser time(preferably less than one second) using Akka Streams?