Unsure How Calling Another Service Works?

[reid-spencer]

I am attempting to do this:

 val tenantService = actionContext.getGrpcClient(classOf[TenantService], "TenantEntity")

in the implementation of an integration test for an action named “Gateway”. The GatewayServiceAction invokes a method on itself and the implementation of that method attempts to find the tenantService as above.

Unfortunately, I get the following symptom during the getGrpcClient call:

WARNING: [Channel<5>: (//TenantEntity)] Failed to resolve name. status=Status{code=UNKNOWN, 
description=TenantEntity: nodename nor servname provided, or not known, cause=null}

The TenantService is an event_sourced_entity defined like this:
API:

service TenantService {
  option(akkaserverless.service) = {
    type: SERVICE_TYPE_ENTITY
    component : "com.yoppworks.coinage.tenant.domain.TenantEntity"
  };

  rpc configureTenant(ConfigureTenant) returns (TenantInfo);
  rpc fetchTenant(FetchTenant) returns (TenantInfo);
}

DOMAIN:

option (akkaserverless.file).event_sourced_entity = {
  name: "TenantEntity"
  entity_type: "tenants"
  state: "TenantState"
  events: [ "TenantConfigured" ]
};

Based on the “calling another service” documentation, and examples shown, I presume the unresolved name is the “name” field from the TenantEntity event_sourced_entity. Using that value doesn’t work and produces the symptom above. Same for the “entity_type” value.

I’m a bit stumped about how to get akkaserverless to find the service I’ve referenced and hoping someone can clue me in.

Also, it isn’t quite clear in the documentation about defining event-sourced-entities what the values in an event_source_entity option refer to because the labels are not defined (just above here) which looks like:

// Describes how this domain relates to an event sourced entity
option (akkaserverless.file).event_sourced_entity = { (3)
  name: "ShoppingCart" (4)
  entity_type: "shopping-cart" (5)
  state: "Cart" (6)
  events: ["ItemAdded", "ItemRemoved"] (7)
};

None of those numbered labels (3-7) are defined below that code sample. I would prepare a PR for that but this issue leads me to believe I don’t know how these options are used.

Full exception trace for the symptom:

2021-11-18 13:03:34,674 INFO  🐳 [gcr.io/akkaserverless-public/akkaserverless-proxy:0.8.1] - Container gcr.io/akkaserverless-public/akkaserverless-proxy:0.8.1 started in PT11.142277S

2021-11-18 13:03:41,385 DEBUG com.yoppworks.coinage.tenant.DefaultTenantServiceClient - monitoring with state IDLE and connectionAttempts 0
Nov 18, 2021 1:03:41 PM io.grpc.internal.ManagedChannelImpl$NameResolverListener handleErrorInSyncContext
WARNING: [Channel<5>: (//TenantEntity)] Failed to resolve name. status=Status{code=UNKNOWN, description=TenantEntity: nodename nor servname provided, or not known, cause=null}
2021-11-18 13:03:41,484 DEBUG com.yoppworks.coinage.tenant.DefaultTenantServiceClient - monitoring with state TRANSIENT_FAILURE and connectionAttempts 0
2021-11-18 13:03:41,494 ERROR com.akkaserverless.javasdk.impl.action.ActionsImpl - Failure during handling of command com.yoppworks.coinage.gateway.GatewayService.createTenant
io.grpc.StatusRuntimeException: UNKNOWN: TenantEntity: nodename nor servname provided, or not known
	at io.grpc.Status.asRuntimeException(Status.java:535)
	at akka.grpc.internal.UnaryCallAdapter.onClose(UnaryCallAdapter.scala:40)
	at io.grpc.internal.DelayedClientCall$DelayedListener$3.run(DelayedClientCall.java:463)
	at io.grpc.internal.DelayedClientCall$DelayedListener.delayOrExecute(DelayedClientCall.java:427)
	at io.grpc.internal.DelayedClientCall$DelayedListener.onClose(DelayedClientCall.java:460)
	at io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:557)
	at io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:69)
	at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInternal(ClientCallImpl.java:738)
	at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:717)
	at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
	at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:133)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:834)


The future returned an exception of type: io.grpc.StatusRuntimeException, with message: UNKNOWN: Unexpected error [595c5ea3-ab42-4d27-b60a-95c0692d22c7].
ScalaTestFailureLocation: com.yoppworks.coinage.gateway.GatewayServiceActionIntegrationSpec at (GatewayServiceActionIntegrationSpec.scala:34)
org.scalatest.exceptions.TestFailedException: The future returned an exception of type: io.grpc.StatusRuntimeException, with message: UNKNOWN: Unexpected error [595c5ea3-ab42-4d27-b60a-95c0692d22c7].
	at org.scalatest.concurrent.ScalaFutures$$anon$1.futureValueImpl(ScalaFutures.scala:333)
	at org.scalatest.concurrent.Futures$FutureConcept.futureValue(Futures.scala:476)
	at org.scalatest.concurrent.Futures$FutureConcept.futureValue$(Futures.scala:475)
	at org.scalatest.concurrent.ScalaFutures$$anon$1.futureValue(ScalaFutures.scala:281)
	at com.yoppworks.coinage.gateway.GatewayServiceActionIntegrationSpec.$anonfun$new$2(GatewayServiceActionIntegrationSpec.scala:34)
	at org.scalatest.OutcomeOf.outcomeOf(OutcomeOf.scala:85)
	at org.scalatest.OutcomeOf.outcomeOf$(OutcomeOf.scala:83)
	at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104)
	at org.scalatest.Transformer.apply(Transformer.scala:22)
	at org.scalatest.Transformer.apply(Transformer.scala:20)
	at org.scalatest.wordspec.AnyWordSpecLike$$anon$3.apply(AnyWordSpecLike.scala:1227)
	at org.scalatest.TestSuite.withFixture(TestSuite.scala:196)
	at org.scalatest.TestSuite.withFixture$(TestSuite.scala:195)
	at org.scalatest.wordspec.AnyWordSpec.withFixture(AnyWordSpec.scala:1879)
	at org.scalatest.wordspec.AnyWordSpecLike.invokeWithFixture$1(AnyWordSpecLike.scala:1225)
	at org.scalatest.wordspec.AnyWordSpecLike.$anonfun$runTest$1(AnyWordSpecLike.scala:1237)
	at org.scalatest.SuperEngine.runTestImpl(Engine.scala:306)
	at org.scalatest.wordspec.AnyWordSpecLike.runTest(AnyWordSpecLike.scala:1237)
	at org.scalatest.wordspec.AnyWordSpecLike.runTest$(AnyWordSpecLike.scala:1219)
	at org.scalatest.wordspec.AnyWordSpec.runTest(AnyWordSpec.scala:1879)
	at org.scalatest.wordspec.AnyWordSpecLike.$anonfun$runTests$1(AnyWordSpecLike.scala:1296)
	at org.scalatest.SuperEngine.$anonfun$runTestsInBranch$1(Engine.scala:413)
	at scala.collection.immutable.List.foreach(List.scala:333)
	at org.scalatest.SuperEngine.traverseSubNodes$1(Engine.scala:401)
	at org.scalatest.SuperEngine.runTestsInBranch(Engine.scala:390)
	at org.scalatest.SuperEngine.$anonfun$runTestsInBranch$1(Engine.scala:427)
	at scala.collection.immutable.List.foreach(List.scala:333)
	at org.scalatest.SuperEngine.traverseSubNodes$1(Engine.scala:401)
	at org.scalatest.SuperEngine.runTestsInBranch(Engine.scala:396)
	at org.scalatest.SuperEngine.runTestsImpl(Engine.scala:475)
	at org.scalatest.wordspec.AnyWordSpecLike.runTests(AnyWordSpecLike.scala:1296)
	at org.scalatest.wordspec.AnyWordSpecLike.runTests$(AnyWordSpecLike.scala:1295)
	at org.scalatest.wordspec.AnyWordSpec.runTests(AnyWordSpec.scala:1879)
	at org.scalatest.Suite.run(Suite.scala:1112)
	at org.scalatest.Suite.run$(Suite.scala:1094)
	at org.scalatest.wordspec.AnyWordSpec.org$scalatest$wordspec$AnyWordSpecLike$$super$run(AnyWordSpec.scala:1879)
	at org.scalatest.wordspec.AnyWordSpecLike.$anonfun$run$1(AnyWordSpecLike.scala:1341)
	at org.scalatest.SuperEngine.runImpl(Engine.scala:535)
	at org.scalatest.wordspec.AnyWordSpecLike.run(AnyWordSpecLike.scala:1341)
	at org.scalatest.wordspec.AnyWordSpecLike.run$(AnyWordSpecLike.scala:1339)
	at com.yoppworks.coinage.gateway.GatewayServiceActionIntegrationSpec.org$scalatest$BeforeAndAfterAll$$super$run(GatewayServiceActionIntegrationSpec.scala:17)
	at org.scalatest.BeforeAndAfterAll.liftedTree1$1(BeforeAndAfterAll.scala:213)
	at org.scalatest.BeforeAndAfterAll.run(BeforeAndAfterAll.scala:210)
	at org.scalatest.BeforeAndAfterAll.run$(BeforeAndAfterAll.scala:208)
	at com.yoppworks.coinage.gateway.GatewayServiceActionIntegrationSpec.run(GatewayServiceActionIntegrationSpec.scala:17)
	at org.scalatest.tools.SuiteRunner.run(SuiteRunner.scala:45)
	at org.scalatest.tools.Runner$.$anonfun$doRunRunRunDaDoRunRun$13(Runner.scala:1322)
	at org.scalatest.tools.Runner$.$anonfun$doRunRunRunDaDoRunRun$13$adapted(Runner.scala:1316)
	at scala.collection.immutable.List.foreach(List.scala:333)
	at org.scalatest.tools.Runner$.doRunRunRunDaDoRunRun(Runner.scala:1316)
	at org.scalatest.tools.Runner$.$anonfun$runOptionallyWithPassFailReporter$24(Runner.scala:993)
	at org.scalatest.tools.Runner$.$anonfun$runOptionallyWithPassFailReporter$24$adapted(Runner.scala:971)
	at org.scalatest.tools.Runner$.withClassLoaderAndDispatchReporter(Runner.scala:1482)
	at org.scalatest.tools.Runner$.runOptionallyWithPassFailReporter(Runner.scala:971)
	at org.scalatest.tools.Runner$.run(Runner.scala:798)
	at org.scalatest.tools.Runner.run(Runner.scala)
	at org.jetbrains.plugins.scala.testingSupport.scalaTest.ScalaTestRunner.runScalaTest2or3(ScalaTestRunner.java:38)
	at org.jetbrains.plugins.scala.testingSupport.scalaTest.ScalaTestRunner.main(ScalaTestRunner.java:25)
Caused by: io.grpc.StatusRuntimeException: UNKNOWN: Unexpected error [595c5ea3-ab42-4d27-b60a-95c0692d22c7]
	at io.grpc.Status.asRuntimeException(Status.java:535)
	at akka.grpc.internal.UnaryCallAdapter.onClose(UnaryCallAdapter.scala:40)
	at io.grpc.internal.DelayedClientCall$DelayedListener$3.run(DelayedClientCall.java:463)
	at io.grpc.internal.DelayedClientCall$DelayedListener.delayOrExecute(DelayedClientCall.java:427)
	at io.grpc.internal.DelayedClientCall$DelayedListener.onClose(DelayedClientCall.java:460)
	at io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:557)
	at io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:69)
	at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInternal(ClientCallImpl.java:738)
	at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:717)
	at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
	at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:133)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)

[reid-spencer]

After I posted the above, I noticed here in the Akkaserverless discussions that release 0.10.0 is out, the documentation updated, and there is a successful way to have an Action call a service, via the “components” accessible in the Action. I refactored my tests, upgraded to 0.10.0 and it “just works”.

[HarveyEllis]

@reid-spencer - so this is running the tests using docker locally?

I ask because I also have this problem locally using java 16, akka serverless 0.10.0 and
gcr.io/akkaserverless-public/akkaserverless-proxy:0.8.1 but get this when running docker compose and testing manually using curl. (I’m basically following the counter set up guide, and adding an action that calls the counter service).

Was the fix just to use TenantEntity as the name in your case, and update to 0.10.0?

I should note, I was able to work around this by adding the following to application.conf:

akka.grpc.client {
  "counter" {
    service-discovery {
      service-name = "counter"
    }
    host = "localhost"
    port = 9000
    use-tls = false
  }
}

See docs here for more details.

[johanandren (Johan Andrén)]

@HarveyEllis if the entity is another component in the same deployed service, you should call it through components.otherEntity.someMethod(request) and not need any addition to application.conf, but if the entity is in another separately deployed service, you should use the actionContext.getGrpcClient(classOf[GrpcServiceInterfaceName], "deployed-name") where the deployed-name is the name you have deployed the other service as in Akka Serverless.

The inter-component call should work automagically when running locally or running integration tests.

A cross service call (just like a regular gRPC call to an external service) will need som specific configuration when running locally and when running integration tests to tell Akka Serverless what service you want to connect to. For example a stub/dummy implementation, or a separate test version of the service.

This you can do with config just like the snippet you shared pointing “counter” to “localhost:9000” when running locally. You will however not want to put it in src/main/resources/application.conf since that will also redirect calls to localhost also when the service is actually deployed and running. Instead put it in src/test/resources/ so that it is only picked up during tests, and/or run with a specific config for running locally.

I think we could improve docs around this, but there is not much to help you figure this out yet.

[HarveyEllis]

@johanandren - thanks for the clarifications.

@johanandren

The inter-component call should work automagically when running locally or running integration tests.

Yep - it does, I was just trying to emulate what was going on if I had multiple services like you say.

@johanandren

You will however not want to put it in src/main/resources/application.conf since that will also redirect calls to localhost also when the service is actually deployed and running. Instead put it in src/test/resources/ so that it is only picked up during tests, and/or run with a specific config for running locally.

Ah yeah - I’d forgot about that, thanks! I hadn’t actually deployed this test example as was just messing about with it locally so hadn’t encountered that. Good to point out for future users though!

[johanandren (Johan Andrén)]

I have created a small Scala two-service sample with some details of how to run multiple services either all locally or mixed with deployed and a local service that may be useful for anyone having problems with this, details and explanations in the README.md here:

akkaserverless-multiple-services-sample/service-two at main ·...

[michallis (Michallis Pashidis)]

Dear,

In Akka SLS - addSbtPlugin(“com.akkaserverless” % “sbt-akkaserverless” % “0.10.6”).
Calling internal persistent entity services, fails (following the guidance here above).

Defining a proto:

service AllocationController {
  option (akkaserverless.codegen) = {
    action: {}
  };

  rpc GetConsumptions(api.GetConsumptionsRequest) returns (api.AllocationResponse) {
    option (google.api.http) = {
      post: "/v1/allocations"
      body: "*"
    };
  };
}

And after generating the action controller:

  override def getConsumptions(cr: GetConsumptionsRequest): Action.Effect[AllocationResponse] = {
    val es = components.energySourceEntity.getEnergySource(GetEnergySourceRequest(cr.sourceId)).execute()
    val effect: Future[Action.Effect[AllocationResponse]] = es.map(_ => effects.reply(AllocationResponse.defaultInstance)).recover(_ => effects.error(s"Failed to get Energy Source ${cr.sourceId}"))
    effects.asyncEffect(effect)
  }

I get the following errors:

[info] {"timestamp":"2022-05-01T07:31:37.652Z","thread":"akkaserverless-akka.actor.default-dispatcher-3","mdc":{"akkaAddress":"akka://akkaserverless","sourceThread":"akkaserverless-akka.actor.default-dispatcher-7","akkaSource":"DefaultEnergySourceServiceClient(akka://akkaserverless)","sourceActorSystem":"akkaserverless","akkaTimestamp":"07:31:37.652UTC"},"logger":"com.enelyzer.ca.api.DefaultEnergySourceServiceClient","message":"monitoring with state IDLE and connectionAttempts 0","context":"default","severity":"DEBUG"}
[info] {"timestamp":"2022-05-01T07:31:37.687Z","thread":"akkaserverless-akka.actor.default-dispatcher-3","mdc":{"akkaAddress":"akka://akkaserverless","sourceThread":"grpc-default-executor-1","akkaSource":"DefaultEnergySourceServiceClient(akka://akkaserverless)","sourceActorSystem":"akkaserverless","akkaTimestamp":"07:31:37.686UTC"},"logger":"com.enelyzer.ca.api.DefaultEnergySourceServiceClient","message":"monitoring with state CONNECTING and connectionAttempts 0","context":"default","severity":"DEBUG"}
[info] {"timestamp":"2022-05-01T07:31:37.838Z","thread":"akkaserverless-akka.actor.default-dispatcher-7","mdc":{"akkaAddress":"akka://akkaserverless","sourceThread":"grpc-default-executor-1","akkaSource":"DefaultEnergySourceServiceClient(akka://akkaserverless)","sourceActorSystem":"akkaserverless","akkaTimestamp":"07:31:37.838UTC"},"logger":"com.enelyzer.ca.api.DefaultEnergySourceServiceClient","message":"monitoring with state TRANSIENT_FAILURE and connectionAttempts 0","context":"default","severity":"DEBUG"}
[info] {"timestamp":"2022-05-01T07:31:38.768Z","thread":"akkaserverless-akka.actor.default-dispatcher-3","mdc":{"akkaAddress":"akka://akkaserverless","sourceThread":"grpc-default-executor-1","akkaSource":"DefaultEnergySourceServiceClient(akka://akkaserverless)","sourceActorSystem":"akkaserverless","akkaTimestamp":"07:31:38.768UTC"},"logger":"com.enelyzer.ca.api.DefaultEnergySourceServiceClient","message":"monitoring with state CONNECTING and connectionAttempts 1","context":"default","severity":"DEBUG"}
[info] {"timestamp":"2022-05-01T07:31:38.769Z","thread":"akkaserverless-akka.actor.default-dispatcher-7","mdc":{"akkaAddress":"akka://akkaserverless","sourceThread":"grpc-default-executor-1","akkaSource":"DefaultEnergySourceServiceClient(akka://akkaserverless)","sourceActorSystem":"akkaserverless","akkaTimestamp":"07:31:38.769UTC"},"logger":"com.enelyzer.ca.api.DefaultEnergySourceServiceClient","message":"monitoring with state TRANSIENT_FAILURE and connectionAttempts 1","context":"default","severity":"DEBUG"}
[info] {"timestamp":"2022-05-01T07:31:40.076Z","thread":"akkaserverless-akka.actor.default-dispatcher-3","mdc":{"akkaAddress":"akka://akkaserverless","sourceThread":"grpc-default-executor-1","akkaSource":"DefaultEnergySourceServiceClient(akka://akkaserverless)","sourceActorSystem":"akkaserverless","akkaTimestamp":"07:31:40.076UTC"},"logger":"com.enelyzer.ca.api.DefaultEnergySourceServiceClient","message":"monitoring with state CONNECTING and connectionAttempts 2","context":"default","severity":"DEBUG"}
[info] {"timestamp":"2022-05-01T07:31:40.078Z","thread":"akkaserverless-akka.actor.default-dispatcher-7","mdc":{"akkaAddress":"akka://akkaserverless","sourceThread":"grpc-default-executor-1","akkaSource":"DefaultEnergySourceServiceClient(akka://akkaserverless)","sourceActorSystem":"akkaserverless","akkaTimestamp":"07:31:40.078UTC"},"logger":"com.enelyzer.ca.api.DefaultEnergySourceServiceClient","message":"monitoring with state TRANSIENT_FAILURE and connectionAttempts 2","context":"default","severity":"DEBUG"}

The persistent entity I’m calling is defined as:

service EnergySourceService {
  option (akkaserverless.codegen) = {
    event_sourced_entity: {
      name: "com.enelyzer.ca.domain.EnergySourceEntity"
      entity_type: "energySource"
      state: "com.enelyzer.ca.domain.EnergySource"
      events: [
        "com.enelyzer.ca.domain.EnergySourceCreated",
        "com.enelyzer.ca.domain.EnergySourceUpdated"
      ]
    }
  };

To be clear the persistent entity is defined in the same service as the function (from action) calling.

[michallis (Michallis Pashidis)]

Just to be clear, this is not a solution on my issue:

Running multiple Akka Serverless value entity services in docker-compose

Ah, thanks for the reply. For reference, my docker-compose.yml looks like this (but with more than 2 Akka SLS services): version: "3" services: gcloud-pubsub-emulator: container_name: "gcloud-pubsub-emulator" image: gcr.io/google.com/cloudsdktool/cloud-sdk:341.0.0 command: gcloud beta emulators pubsub start --project=test --host-port=0.0.0.0:8085 ports: - "8085:8085" networks: - akka-net akka-sls-1-proxy: container_name: "akka-sls-1-proxy" image: g…

(forward indeed looses the possibility of transforming the response)

[johanandren (Johan Andrén)]

We recently discovered a bug around inter-component calls that I think that could explain the issue.

Does it work as expected when run locally?

[michallis (Michallis Pashidis)]

Hi Johan,
It’s not working locally, but it was before in my opinion.

best regards,
Michallis