Lazy eval of ProducerMessage.multi

Akka Akka Streams & Alpakka
,
1

Hi,
I’m looking at transitioning an existing codebase to Alpakka Kafka and one of the requirements is that a single Kafka message can result in the output of 100k+ of Kafka messages. It also requires at-least-once delivery semantics. This is currently done with Scala views so that the producer messages are produced lazily and don’t consume vast amounts of memory. Looking at Alpakka, the ProduceMessage.multi takes an immutable Seq which is not consistent with view. I could use a Stream which would use lazy evaluation but would still have the problem with the memory footprint.

Is there any way to to solve this problem?

2

Hi @Elston15

You seem to have an interesting use case with producing 100k+ messages from one incoming message.

With the current implementation of ProducerMessage.multi the messages can’t be created lazily.
As Alpakka Kafka cross-compiles for Scala 2.11, 2.12 and 2.13 I believe collection views are not a viable option as they work differently since 2.13.

If you have a suggestion how we could improve the API to solve this, feel free to open a pull request.

Cheers,
Enno.