I have been trying to use Alpakka S3 to perform a rather straightforward operation:
- List all files on a given bucket
- Download all of them and parse content
It seem to me that the resulting code is unwieldy complicated with nested Sources, is there a way to simplify / flatten the structure somehow? Am I missing something obvious?
S3.download is now a Source[Option[Source[ByteString]]], is there an easy way to flatten it out into just a Source[ByteString] since I already know that the file exists.
So e.g. I would like to flatten:
val file: Source[Source[Option[(Source[ByteString, NotUsed], ObjectMetadata)], NotUsed], NotUsed] = S3.listBucket(conf.bucketName, Some(conf.prefixFilter)) .map(r => S3.download(r.bucketName, r.key))
to something like:
Any thoughts are welcomed,