Alpakka and S3 truncating downloaded files

Hello,

I have a simple piece of code, based on alpakka examples, which should download some file from S3 for further processing:

S3.download(bucket, file)
  .runWith(Sink.head)
  .flatMap {
    case Some((data, _)) =>
      data.map(_.utf8String).runWith(Sink.head).map(Some.apply)
    case None =>
      Future.successful(None)
    }

The problem is that the file content is getting truncated, the file size from the ObjectMetadata is correct it has ~2M it isn’t a huge file.

What I noticed is, when I use Sink.head the file content is from the beginning to middle if I change it to Sink.last it is from the middle to the end. Am I getting chunks from the file but why are they not seem to be streamed?

Can’t figure out what’s happening and how to fix this.

Thanks

Please disregard my question I’ve found the solution myself. In the end it was very clear…

Just need to replace:
data.map(_.utf8String).runWith(Sink.head).map(Some.apply)

with:
data.map(_.utf8String).runWith(Sink.seq).map(_.mkString).map(Some.apply)

accumullating all the chunks from the file.

Thanks

1 Like