Running multiple Akka Serverless value entity services in docker-compose

Akka Serverless
#1

Hi,

This question is sort of related to this thread, which helped me out already.

We’re developing a number of Akka Serverless services (v0.10.6, in Scala), along with some Play Framework & Lagom services, and we want to be able to run all services locally during development. Using the information from the thread mentioned above, I was able to get all of our Akka SLS services with event-sourced entities running locally via a docker-compose file.

However, none of our value entity services work, with requests failing with the following stacktrace:

io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
  at io.grpc.Status.asRuntimeException(Status.java:535)
  at akka.grpc.internal.UnaryCallAdapter.onClose(UnaryCallAdapter.scala:40)
  at io.grpc.internal.DelayedClientCall$DelayedListener$3.run(DelayedClientCall.java:463)
  at io.grpc.internal.DelayedClientCall$DelayedListener.delayOrExecute(DelayedClientCall.java:427)
  at io.grpc.internal.DelayedClientCall$DelayedListener.onClose(DelayedClientCall.java:460)
  at io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:562)
  at io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:70)
  at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInternal(ClientCallImpl.java:743)
  at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:722)
  at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
  at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:133)
  at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
  at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
  at java.base/java.lang.Thread.run(Unknown Source)
Caused by: io.grpc.netty.shaded.io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: Connection refused: localhost/127.0.0.1:9000
Caused by: java.net.ConnectException: finishConnect(..) failed: Connection refused
  at io.grpc.netty.shaded.io.netty.channel.unix.Errors.newConnectException0(Errors.java:155)
  at io.grpc.netty.shaded.io.netty.channel.unix.Errors.handleConnectErrno(Errors.java:128)
  at io.grpc.netty.shaded.io.netty.channel.unix.Socket.finishConnect(Socket.java:278)
  at io.grpc.netty.shaded.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.doFinishConnect(AbstractEpollChannel.java:710)
  at io.grpc.netty.shaded.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:687)
  at io.grpc.netty.shaded.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:567)
  at io.grpc.netty.shaded.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:470)
  at io.grpc.netty.shaded.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378)
  at io.grpc.netty.shaded.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
  at io.grpc.netty.shaded.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
  at io.grpc.netty.shaded.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
  at java.base/java.lang.Thread.run(Unknown Source)

The services in question do work fine in the cloud, as well as when running them individually as described in the documentation. After enabling debug logging and a bit of code diving, I’m guessing this is related to this issue.

Am i misinterpreting the error/looking in the wrong places, or are there any workarounds available? Thanks in advance.

#2

I think your guess about the linked issue is correct, this is probably a cross component call not knowing on which port you are actually running the service itself. Unfortunately I don’t know of a simple workaround until we have a solution for the issue.

You could perhaps try to do the docker compose set up in some way that does not forward multiple proxies to localhost but instead talk to each other on different container IPs.

#3

Ah, thanks for the reply.

For reference, my docker-compose.yml looks like this (but with more than 2 Akka SLS services):

version: "3"

services:
  gcloud-pubsub-emulator:
    container_name: "gcloud-pubsub-emulator"
    image: gcr.io/google.com/cloudsdktool/cloud-sdk:341.0.0
    command: gcloud beta emulators pubsub start --project=test --host-port=0.0.0.0:8085
    ports:
      - "8085:8085"
    networks:
      - akka-net

  akka-sls-1-proxy:
    container_name: "akka-sls-1-proxy"
    image: gcr.io/akkaserverless-public/akkaserverless-proxy:latest
    command: -Dconfig.resource=dev-mode.conf -Dakkaserverless.proxy.eventing.support=google-pubsub-emulator
    ports:
      - "10000:9000"
    networks:
      - akka-net
    environment:
      USER_FUNCTION_HOST: en-akkasls-1
      USER_FUNCTION_PORT: 8080
      PUBSUB_EMULATOR_HOST: gcloud-pubsub-emulator
    depends_on:
      - gcloud-pubsub-emulator
      - akka-sls-1

  akka-sls-1:
    image: akka-sls-1:latest
    container_name: "akka-sls-1"
    networks:
      - akka-net
    environment:
      JAVA_OPTS: "-Dconfig.file=/opt/docker/conf/application.conf" # Sets akkaserverless.user-function-interface=0.0.0.0
    volumes:
      - "./docker/akkasls/app.conf:/opt/docker/conf/application.conf" 
    depends_on:
      - gcloud-pubsub-emulator

  akka-sls-2-proxy:
    container_name: "akka-sls-2-proxy"
    image: gcr.io/akkaserverless-public/akkaserverless-proxy:latest
    command: -Dconfig.resource=dev-mode.conf -Dakkaserverless.proxy.eventing.support=google-pubsub-emulator
    ports:
      - "10001:9000"
    networks:
      - akka-net
    environment:
      USER_FUNCTION_HOST: en-akkasls-1
      USER_FUNCTION_PORT: 8080
      PUBSUB_EMULATOR_HOST: gcloud-pubsub-emulator
    depends_on:
      - gcloud-pubsub-emulator
      - akka-sls-2

  akka-sls-2:
    image: akka-sls-2:latest
    container_name: "akka-sls-2"
    networks:
      - akka-net
    environment:
      JAVA_OPTS: "-Dconfig.file=/opt/docker/conf/application.conf" # Sets akkaserverless.user-function-interface=0.0.0.0
    volumes:
      - "./docker/akkasls/app.conf:/opt/docker/conf/application.conf"
    depends_on:
      - gcloud-pubsub-emulator

networks:
  akka-net:
    driver: bridge

Not sure how to go about trying your suggestion.

But with some further testing, I changed the code of one of my Actions from the example in the documentation:

  override def initializeCart(newCart: NewCart): Action.Effect[NewCartCreated] = {
    val cartId = UUID.randomUUID().toString

    val created: Future[Empty] =
      components.shoppingCart.create(CreateCart(cartId)).execute()

    val effect: Future[Action.Effect[NewCartCreated]] =
      created.map(_ => effects.reply(NewCartCreated(cartId)))
        .recover(_ => effects.error("Failed to create cart, please retry"))

    effects.asyncEffect(effect)
  }

to a forward call like in the code below, and for now everything seems to be working :crossed_fingers:

  override def initializeCart(newCart: NewCart): Action.Effect[NewCartCreated] = {
    val cartId = UUID.randomUUID().toString
    effects.forward(components.shoppingCart.create(CreateCart(cartId)))
  }
Unsure How Calling Another Service Works?
#4

The reason why forwarding works is that it is a part of the proxy protocol response rather than a separate gRPC call to another component and therefore doesn’t need to know where the service can be found.

Great if you can do that instead, what you lose compared to the async component call is the capability to map the response message if that is needed.