Optimizing akka stream throughput with multiple slow flows

I’m using akka 2.6.4.

I’m trying to figure out the right way of optimizing an akka stream for throughput. The stream in question will have several flows, where each flow will typically call into a separate web service. These calls will typically batch up several messages using the groupedWithin operator, and send the result downstream using flatMapConcat.

Example:

  • source
  • Flow [groupedwithin(10, 1.second) - call some service with batch - zip results - flatMapConcat]
  • Flow [groupedwithin(50, 1.second) - call some other service with batch - zip results - flatMapConcat]
  • Flow [groupedwithin(30, 1.second) - call some third service with batch - zip results - flatMapConcat]
  • sink

Given that we have several of these flow in our stream we want to figure out which of the flows that is the slowest one. The backpressure feature of akka (which is great) makes figuring out which flow of the stream more tricky, since the entires stream slows down to the slowest flow in the entire stream.

Is there some established pattern for figuring out which part of the stream is the slowest, given that the entire stream is backpressured?


Things I’ve looked into:

https://github.com/ruippeixotog/akka-stream-mon contains some great tools for measuring the throughput and also latency for each individual step. The throughput tool only measures the backpressued throughput, and doesn’t allow me to figure out which part of the stream is the slowest. The latency tool would have been great if it weren’t for that fact that I’m batching in each flow, so the latency number doesn’t really tell me if that flow is the slow one. The only thing I care about is optimizing for throughput.

Came across this question: https://groups.google.com/forum/#!topic/akka-user/LLXOooUwUIE which asks about monitorable buffers. I believe that would help me pin-point which buffer is full/empty, and that could help me determine where in the stream it slows down. The reply talks about some Lightbend Reactive Monitoring, but it’s from back in 2017, so I don’t know if anything came of that…