Start consuming Source(s) elements before Sink pull ( flow)


I’m using Alpakka flow.
(As reminder, here is the signature of the Flow is Flow[(ArchiveMetadata, Source[ByteString, Any]), ByteString, NotUsed]).

The flow is consuming inner sources one by one.
I would like to start consuming others inner sources in order to speed up the process (at least until the buffer reach a given size).

Code example:

val objectKeys: Seq[String] = ...

val zipObjects: Seq[(ArchiveMetadata, Source[ByteString, NotUsed])] = { obj =>  
  val filename = trimPrefixFromKey(obj)
  ArchiveMetadata(filename) ->"myBucket", obj)
    .flatMapConcat {  
      case Some((dataSource, _)) =>
          .flatMapConcat { case (head, tail) =>  
              .wireTap(_ =>"Start consuming {}", obj))  
          /* How to start consuming let's say 50 MB ??? */ 
      case None => Source.single(ByteString.empty)

val s3Sink: Sink[ByteString, Future[MultipartUploadResult]] = ...

val uploadResultFuture = Source(zipObjects)  
1 Like

Same question

Created a ticket (and PR) => Add supports for Byte-Range Fetches when downloading from S3 · Issue #2981 · akka/alpakka · GitHub

Buffering as described above is pointless.
Preferred improving download speed directly.