I discovered Scala / Akka few months ago and figured out Actors / Reactive streams could be the right toolbox to create an IoT data integration application.
To summarize it briefly, GB of IoT data are streamed continuously to a Kafka cluster.
All this data needs to be aggregated per batch in order to be transformed in a fast read storage format.
I figured the actor model (batch jobs notifications) and akka reactive streams (for guaranteed delivery and back-pressure) should be the right tools :
Consumer : consumes a batch of data and send a BatchNotification to a JobManager with guaranteed delivery
JobManager : build appropriate jobs and send it all to a Worker Master actor which broadcast JobAvailable to a pool of Worker actor.
I figured out how to consume data from kafka (using “akka.kafka.scaladsl.Consumer”) and send a job to the JobManager using “toMat” (I have to admit it’s still unclear what these materialized views exactly mean).
One thing which remains to be done, and which I have trouble implementing, is implementing back-pressure and ACK.
In other words, the consumer should wait for an ACK from JobManager before commiting Kafka offsets and sending a new job to it.
I’ve no idea how to implement this, and can’t find any example of this.
Is this easily implementable using Akka actors / streams ?
Any hint on the design pattern, doc and package to use ?