Thanks for the info. Using
mapAsync is where I ended up but did give the Source with context a try.
Essentially what I’m trying to do is consume from kafka (partitioned source from alpakka-kafka), groupBy within one of these partitioned sources and write to hdfs using alpakka-hdfs across many substreams. Then merge these substreams to form a stream of
OutgoingMessages. The reason I want in-order is to store all the rotation events in a file with the offsets in order (for recovery).
So I’ve ended up keeping alpakka-kafka but not using alpakka-hdfs and using my own writer which is shame.
An approach I was trying was to modify the
SubFlowImpl.MergeBack to use my own
zipWithIndex on the flow before the
groupBy. Then the
OrderedFlattenMerge would only push an element if the index is lastSeen+1 or if all the substreams have an element buffered and so pushes the smallest index. What do you think?