You can achieve parallelism a number of different ways with Alpakka Kafka. The standard way is to run multiple applications that use the same Kafka consumer group id and topic subscription.
You may also achieve parallelism within each application by using the
mapAsync operator or the “partitioned”
mapAsync will give you a “threadpool”-like ability to process messages you consume. For CPU-bound processing you can should the
parallelism argument of
mapAsync to the number of cores available to the app, for IO-bound operations you should choose a value that works best for you (i.e. max number of connections you want to make to a networked service).
Sources provide a way of processing messages from different partitions separately, instead of funnelling messages of all partitions into one stream. You may use the various sub stream operators here to achieve parallelism.
As for what
Source to choose, it’s probably best to start with
plainSource if you don’t need to commit offsets back to Kafka.
Hope that helps!