I have an Akka Stream application written with following topology:
=> There are 3 similar sources in parallel which generates messages.
=> All 3 sources gets merged via Merge component.
=> Further within each source the source is created by 2 different stages(all 3 sources are similar except that they are connected to different topics):
1. In Stage 1 a SoureShape which has some custom sophisticated logic and emits an iterator of messages
2. In Stage 2 I am using “flatMapConcat” to emit flattened messages out of iterator.
=> The code looks similar to:
.flatMapConcat(iter => Source.fromIterator(() => iter))
=> “jobProductProviderSourceGraph” emits Iterator[Message] which it pulls from it’s topic
The problem I’m facing is when one of the sources(out of 3) starts emitting messages then until that source runs out of steam of messages the other 2 sources are not emitting any messages despite having messages to be streamed in the middle.
To explain further with timelines:
- Source 1, 2 and 3 are connected to 3 different topics
- At time t1 source 1 has available messages in its topic which it starts emitting until time t1+100
- At time t1+10 source 2 and source 3 have available messages in their topics but doesn’t emit.
Is the “flatMapConcat” separating the source into 2 different stages in the reactive stream in such a way that the demand is not even going to “jobProductProviderSourceGraph” once messages are emitted from Iterator Source?
Can someone please help in making me understand the fundamentals here? Also, please suggest an approach to have the sources emit messages in parallel.