Problem applying custom S3Settings


#1

Hi,

I have a problem applying custom S3 settings to a S3 stream. The listBucket stream in the code snippet below is perfectly working using the specified custom credentials and region provider. But the download stream seems to fall back on the Amazon DefaultProviderChain. If I am setting the environment variables AWS_REGION, AWS_ACCESS_KEY and AWS_SECRET_ACCESS_KEY, I can can download the testfile, otherwise not and I get the exception:

[ERROR] [01/20/2019 06:11:26.859] [test-akka.actor.default-dispatcher-15] [akka://test/system/StreamSupervisor-0/flow-1-0-headSink] Error during preStart in [akka.stream.alpakka.s3.impl.SetupSourceStage@3d2bb897]: Unable to load region information from any provider in the chain
com.amazonaws.SdkClientException: Unable to load region information from any provider in the chain
at com.amazonaws.regions.AwsRegionProviderChain.getRegion(AwsRegionProviderChain.java:59)
at akka.stream.alpakka.s3.impl.HttpRequests$.s3Request(HttpRequests.scala:148)
at akka.stream.alpakka.s3.impl.HttpRequests$.getDownloadRequest(HttpRequests.scala:59)
ā€¦

I tried to apply the S3Settings in almost all code positions I can think of. What I am doing wrong?

Thanks in advance
Lay

public class SourceS3Test {
  @Test
  public void testListBucketAndDownload() throws InterruptedException, ExecutionException {
    ActorSystem actorSystem = ActorSystem.create("test");
    
    Materializer materializer = ActorMaterializer.create(actorSystem);

    AWSCredentialsProvider credentialsProvider =
            new AWSStaticCredentialsProvider(new BasicAWSCredentials("XXXX", "YYYY"));

    AwsRegionProvider regionProvider = new AwsRegionProvider() {
      @Override
      public String getRegion() throws SdkClientException {
        return "eu-central-1";
      }
    };

    S3Settings s3Settings = S3Ext.get(actorSystem).settings()
            .withCredentialsProvider(credentialsProvider)
            .withS3RegionProvider(regionProvider);

    S3.listBucket("testbucket", Option.apply("test"))
            .withAttributes(S3Attributes.settings(s3Settings))
            .to(Sink.foreach(e -> System.out.println(e.getKey())))
            .run(materializer);
    
    Source<Optional<Pair<Source<ByteString, NotUsed>, ObjectMetadata>>,NotUsed> sourceAndMeta =
            S3.download("testbucket","test/testfile.txt")
                    .withAttributes(S3Attributes.settings(s3Settings));

    CompletionStage<Optional<Pair<Source<ByteString,NotUsed>, ObjectMetadata>>> sourceCompletionStage =
            sourceAndMeta
                    .withAttributes(S3Attributes.settings(s3Settings))
                    .toMat(Sink.head(), Keep.right())
                    .run(materializer);

    Optional<Pair<Source<ByteString,NotUsed>, ObjectMetadata>> optionalPair =
            sourceCompletionStage.toCompletableFuture().get();

    Pair<Source<ByteString, NotUsed>, ObjectMetadata> pair = optionalPair.get();

    Source<ByteString, NotUsed> data = pair.first();

    ObjectMetadata metadata = pair.second();

    CompletionStage<String> stringCompletionStage = data.map(ByteString::utf8String)
            .withAttributes(S3Attributes.settings(s3Settings))
            .toMat(Sink.head(), Keep.right())
            .run(materializer);

    System.out.println(stringCompletionStage.toCompletableFuture().get());

    Thread.sleep(10000);
  }
}

(Martynas Mickevičius) #2

Hi. This could be a bug in the refactored settings handling. Iā€™ll take a look and report back.


(Martynas Mickevičius) #3

This is actually a bug for which I created https://github.com/akka/alpakka/issues/1443


#4

Hi Martynas, thanks for your quick resonse! So I will wait for the 1.0-M3 milestone version and will use the AwsDefaultProviderChain in the meanwhile.

Regards,
Lay


(Martynas Mickevičius) #5

When the fix is merged there will be a snapshot version published by Travis. If you can, please try that one out and verify that it works as expected in your case.


#6

Ok, I will get that version and test it.


(Martynas Mickevičius) #7

The fix has been merged and a snapshot version 1.0-M2+17-4b069efb is already in the repository.


#8

Hi, I can confirm that the snapshot fixes the problem and now also in the download case the custom S3Settings are used.

Thanks for this quick fix!
Lay