Java Kickstart on Ubuntu Linux 16.04 LTS / 20.04 LTS

Trying to run the Java Kickstart on Ubuntu 16.04 LTS, but can’t seem to connect to the app via the proxy. The logs all look good for both Docker compose and mvn compile exec:java. Per the README I see this:

Running Locally

On Linux this requires Docker 20.10 or later (Support host.docker.internal in dockerd on Linux by arkodg · Pull Request #40007 · moby/moby · GitHub), or for a USER_FUNCTION_HOST environment variable to be set manually.

docker-compose -f docker-compose.yml -f docker-compose.linux.yml up

However, I’m not able to connect with grpcurl and the request times out.

From my previous notes to run my last project I had to do this:

docker network create -d bridge akkasls-orders
docker run -d --name orders --hostname userfunction --network akkasls-orders <your-dockerhub-username>/orders:ex4 (image name)>
docker run -d --name orders-proxy --network akkasls-orders -p 9000:9000 --env USER_FUNCTION_HOST=userfunction cloudstateio/cloudstate-proxy-dev-mode:0.5.1-98-a596eae7

Note: I have not tried to set USER_FUNCTION_HOST, but will try to adapt to get it running.

For reference:

Client: Docker Engine - Community
 Version:           20.10.7
 API version:       1.41
 Go version:        go1.13.15
 Git commit:        f0df350
 Built:             Wed Jun  2 11:56:47 2021
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true

Server: Docker Engine - Community
 Engine:
  Version:          20.10.7
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.13.15
  Git commit:       b0f5bc3
  Built:            Wed Jun  2 11:54:58 2021
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.4.6
  GitCommit:        d71fcd7d8303cbf684402823e425e9dd2e99285d
 runc:
  Version:          1.0.0-rc95
  GitCommit:        b9ee9c6314599f1b4a7f497e1f1f856fe433d3b7
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

On Ubuntu 20.04 LTS, with the same version of Docker, etc…

docker-compose -f docker-compose.yml -f docker-compose.linux.yml up

the proxy fails with these two repeating errors:

akka-serverless-proxy_1   | {"timestamp":"2021-06-29T23:48:42.124Z","thread":"akkaserverless-proxy-akka.actor.default-dispatcher-3","mdc":{"akkaAddress":"akka://akkaserverless-proxy@172.19.0.3:25520","sourceThread":"akkaserverless-proxy-akka.actor.default-dispatcher-10","akkaSource":"akka://akkaserverless-proxy@172.19.0.3:25520/user/server-manager-supervisor/server-manager","sourceActorSystem":"akkaserverless-proxy","akkaTimestamp":"23:48:42.124UTC"},"logger":"com.akkaserverless.proxy.DiscoveryManager","message":"Connecting to host.docker.internal:8080 to discover the user function specification","context":"default","severity":"INFO"}
akka-serverless-proxy_1   | {"timestamp":"2021-06-29T23:48:42.138Z","thread":"akkaserverless-proxy-akka.actor.default-dispatcher-8","mdc":{"akkaAddress":"akka://akkaserverless-proxy@172.19.0.3:25520","sourceThread":"akkaserverless-proxy-akka.actor.default-dispatcher-3","akkaSource":"akka://akkaserverless-proxy/user/server-manager-supervisor/server-manager","sourceActorSystem":"akkaserverless-proxy","akkaTimestamp":"23:48:42.137UTC"},"logger":"akka.actor.OneForOneStrategy","message":"UNAVAILABLE: io exception","context":"default","exception":"io.grpc.StatusRuntimeException: UNAVAILABLE: io exception\n\tat io.grpc.Status.asRuntimeException(Status.java:534)\n\tat akka.grpc.internal.UnaryCallAdapter.onClose(UnaryCallAdapter.scala:40)\n\tat io.grpc.internal.DelayedClientCall$DelayedListener$3.run(DelayedClientCall.java:464)\n\tat io.grpc.internal.DelayedClientCall$DelayedListener.delayOrExecute(DelayedClientCall.java:428)\n\tat io.grpc.internal.DelayedClientCall$DelayedListener.onClose(DelayedClientCall.java:461)\n\tat io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:553)\n\tat io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:68)\n\tat io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInternal(ClientCallImpl.java:739)\n\tat io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:718)\n\tat io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)\n\tat io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)\n\tat java.base/java.lang.Thread.run(Unknown Source)\nCaused by: io.grpc.netty.shaded.io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: Connection refused: host.docker.internal/172.17.0.1:8080\nCaused by: java.net.ConnectException: finishConnect(..) failed: Connection refused\n\tat io.grpc.netty.shaded.io.netty.channel.unix.Errors.throwConnectException(Errors.java:124)\n\tat io.grpc.netty.shaded.io.netty.channel.unix.Socket.finishConnect(Socket.java:251)\n\tat io.grpc.netty.shaded.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.doFinishConnect(AbstractEpollChannel.java:672)\n\tat io.grpc.netty.shaded.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:649)\n\tat io.grpc.netty.shaded.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:529)\n\tat io.grpc.netty.shaded.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:465)\n\tat io.grpc.netty.shaded.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378)\n\tat io.grpc.netty.shaded.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)\n\tat io.grpc.netty.shaded.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)\n\tat io.grpc.netty.shaded.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)\n\tat java.base/java.lang.Thread.run(Unknown Source)\n","severity":"ERROR"}```

For 16.04 LTS, with the app running locally with mvn compile exec:java I can see these two final log messages:

[INFO] --- exec-maven-plugin:3.0.0:java (default-cli) @ geohash ---
{"timestamp":"2021-07-01T20:52:12.614Z","thread":"com.akkaslshackathon.Main.main()","logger":"com.akkaslshackathon.Main","message":"starting the Akka Serverless service","context":"default","severity":"INFO"}
{"timestamp":"2021-07-01T20:52:13.209Z","thread":"akkaserverless-akka.actor.default-dispatcher-3","logger":"akka.event.slf4j.Slf4jLogger","message":"Slf4jLogger started","context":"default","severity":"INFO"}

Then I do an inspect on the proxy, which shows it’s bound to port 9000:

$ docker inspect akkaserverless-geo-fence_akka-serverless-proxy_1
[
    {
        "Id": "9df066fd88561b95b53789a15d3768afc422ddf6613e5c372c87b846e0bab09f",
        "Created": "2021-07-01T20:35:14.17355975Z",
        "Path": "/opt/docker/bin/akkaserverless-proxy-core",
        "Args": [
            "-Dconfig.resource=dev-mode.conf",
            "-Dakkaserverless.proxy.eventing.support=google-pubsub-emulator"
        ],
        "State": {
            "Status": "running",
            "Running": true,
            "Paused": false,
            "Restarting": false,
            "OOMKilled": false,
            "Dead": false,
            "Pid": 26930,
            "ExitCode": 0,
            "Error": "",
            "StartedAt": "2021-07-01T20:35:16.007596652Z",
            "FinishedAt": "0001-01-01T00:00:00Z"
        },
        "Image": "sha256:a7118624ffdd74f64ff606357518623f3c69ae9296f8bd9f85cd7636daff1a4d",
        "ResolvConfPath": "/var/lib/docker/containers/9df066fd88561b95b53789a15d3768afc422ddf6613e5c372c87b846e0bab09f/resolv.conf",
        "HostnamePath": "/var/lib/docker/containers/9df066fd88561b95b53789a15d3768afc422ddf6613e5c372c87b846e0bab09f/hostname",
        "HostsPath": "/var/lib/docker/containers/9df066fd88561b95b53789a15d3768afc422ddf6613e5c372c87b846e0bab09f/hosts",
        "LogPath": "/var/lib/docker/containers/9df066fd88561b95b53789a15d3768afc422ddf6613e5c372c87b846e0bab09f/9df066fd88561b95b53789a15d3768afc422ddf6613e5c372c87b846e0bab09f-json.log",
        "Name": "/akkaserverless-geo-fence_akka-serverless-proxy_1",
        "RestartCount": 0,
        "Driver": "overlay2",
        "Platform": "linux",
        "MountLabel": "",
        "ProcessLabel": "",
        "AppArmorProfile": "docker-default",
        "ExecIDs": null,
        "HostConfig": {
            "Binds": [],
            "ContainerIDFile": "",
            "LogConfig": {
                "Type": "json-file",
                "Config": {}
            },
            "NetworkMode": "akkaserverless-geo-fence_default",
            "PortBindings": {
                "9000/tcp": [
                    {
                        "HostIp": "",
                        "HostPort": "9000"
                    }
                ]
            },
            "RestartPolicy": {
                "Name": "",
                "MaximumRetryCount": 0
            },
            "AutoRemove": false,
            "VolumeDriver": "",
            "VolumesFrom": [],
            "CapAdd": null,
            "CapDrop": null,
            "CgroupnsMode": "host",
            "Dns": null,
            "DnsOptions": null,
            "DnsSearch": null,
            "ExtraHosts": [
                "host.docker.internal:host-gateway"
            ],
            "GroupAdd": null,
            "IpcMode": "shareable",
            "Cgroup": "",
            "Links": null,
            "OomScoreAdj": 0,
            "PidMode": "",
            "Privileged": false,
            "PublishAllPorts": false,
            "ReadonlyRootfs": false,
            "SecurityOpt": null,
            "UTSMode": "",
            "UsernsMode": "",
            "ShmSize": 67108864,
            "Runtime": "runc",
            "ConsoleSize": [
                0,
                0
            ],
            "Isolation": "",
            "CpuShares": 0,
            "Memory": 0,
            "NanoCpus": 0,
            "CgroupParent": "",
            "BlkioWeight": 0,
            "BlkioWeightDevice": null,
            "BlkioDeviceReadBps": null,
            "BlkioDeviceWriteBps": null,
            "BlkioDeviceReadIOps": null,
            "BlkioDeviceWriteIOps": null,
            "CpuPeriod": 0,
            "CpuQuota": 0,
            "CpuRealtimePeriod": 0,
            "CpuRealtimeRuntime": 0,
            "CpusetCpus": "",
            "CpusetMems": "",
            "Devices": null,
            "DeviceCgroupRules": null,
            "DeviceRequests": null,
            "KernelMemory": 0,
            "KernelMemoryTCP": 0,
            "MemoryReservation": 0,
            "MemorySwap": 0,
            "MemorySwappiness": null,
            "OomKillDisable": false,
            "PidsLimit": null,
            "Ulimits": null,
            "CpuCount": 0,
            "CpuPercent": 0,
            "IOMaximumIOps": 0,
            "IOMaximumBandwidth": 0,
            "MaskedPaths": [
                "/proc/asound",
                "/proc/acpi",
                "/proc/kcore",
                "/proc/keys",
                "/proc/latency_stats",
                "/proc/timer_list",
                "/proc/timer_stats",
                "/proc/sched_debug",
                "/proc/scsi",
                "/sys/firmware"
            ],
            "ReadonlyPaths": [
                "/proc/bus",
                "/proc/fs",
                "/proc/irq",
                "/proc/sys",
                "/proc/sysrq-trigger"
            ]
        },
        "GraphDriver": {
            "Data": {
                "LowerDir": "/var/lib/docker/overlay2/825031c286f13e5b07463751ea89f54ded7b41405388047ce4014271414c84e6-init/diff:/var/lib/docker/overlay2/b2ed94b6573f57ed4fda9d7420be99abe48aaff25482feac228a4d8db13168cd/diff:/var/lib/docker/overlay2/fcb77144a17913f857f6a85fa0f427dc8f92ce1760fcee73d189a7ba5b7a6764/diff:/var/lib/docker/overlay2/3e9ef786af4df5c3d8a9b968925c0bf7be8d5fca4171dc145ea635cda2d9bed9/diff:/var/lib/docker/overlay2/3fbfa57c8fd3994d4c2eaa0b608849a22fa2c8b171c4bd150a8c134483a3afa8/diff:/var/lib/docker/overlay2/1ce517e52648446fa746fc48b384e6745218e5b079b1bfda675978136d88a823/diff:/var/lib/docker/overlay2/179ca282bfe7579c8d99c4ebde0132b26c146f7bf097184f58e43ac970304b6f/diff:/var/lib/docker/overlay2/51b8ea8a0ce8d77b57a18a25208a48e90c6380db163edf4afa2a3c256170cb1b/diff:/var/lib/docker/overlay2/e3eb1fd355f1c67d352a92e8ad6d733d2159d74c28427286ddb3932874c6dc86/diff",
                "MergedDir": "/var/lib/docker/overlay2/825031c286f13e5b07463751ea89f54ded7b41405388047ce4014271414c84e6/merged",
                "UpperDir": "/var/lib/docker/overlay2/825031c286f13e5b07463751ea89f54ded7b41405388047ce4014271414c84e6/diff",
                "WorkDir": "/var/lib/docker/overlay2/825031c286f13e5b07463751ea89f54ded7b41405388047ce4014271414c84e6/work"
            },
            "Name": "overlay2"
        },
        "Mounts": [],
        "Config": {
            "Hostname": "9df066fd8856",
            "Domainname": "",
            "User": "1001:0",
            "AttachStdin": false,
            "AttachStdout": false,
            "AttachStderr": false,
            "ExposedPorts": {
                "9000/tcp": {}
            },
            "Tty": false,
            "OpenStdin": false,
            "StdinOnce": false,
            "Env": [
                "USER_FUNCTION_HOST=host.docker.internal",
                "USER_FUNCTION_PORT=8080",
                "PATH=/opt/java/openjdk/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
                "container=oci",
                "LANG=en_US.UTF-8",
                "LANGUAGE=en_US:en",
                "LC_ALL=en_US.UTF-8",
                "JAVA_VERSION=jdk-11.0.8+10",
                "JAVA_HOME=/opt/java/openjdk"
            ],
            "Cmd": [
                "-Dconfig.resource=dev-mode.conf",
                "-Dakkaserverless.proxy.eventing.support=google-pubsub-emulator"
            ],
            "Image": "gcr.io/akkaserverless-public/akkaserverless-proxy:0.7.0-beta.9",
            "Volumes": null,
            "WorkingDir": "/opt/docker",
            "Entrypoint": [
                "/opt/docker/bin/akkaserverless-proxy-core"
            ],
            "OnBuild": null,
            "Labels": {
                "architecture": "x86_64",
                "build-date": "2020-09-01T19:43:46.041620",
                "com.docker.compose.config-hash": "c8b3471d46d5939afb5924acefa60afed88da5c646b82303cf4a4840aac1c738",
                "com.docker.compose.container-number": "1",
                "com.docker.compose.oneoff": "False",
                "com.docker.compose.project": "akkaserverless-geo-fence",
                "com.docker.compose.service": "akka-serverless-proxy",
                "com.docker.compose.version": "1.22.0",
                "com.redhat.build-host": "cpt-1008.osbs.prod.upshift.rdu2.redhat.com",
                "com.redhat.component": "ubi8-container",
                "com.redhat.license_terms": "https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI",
                "description": "For more information on this image please see https://github.com/AdoptOpenJDK/openjdk-docker/blob/master/README.md",
                "distribution-scope": "public",
                "io.k8s.description": "The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly.",
                "io.k8s.display-name": "Red Hat Universal Base Image 8",
                "io.openshift.expose-services": "",
                "io.openshift.tags": "base rhel8",
                "maintainer": "Red Hat, Inc.",
                "name": "AdoptOpenJDK Java",
                "release": "11",
                "run": "docker run --rm -ti <image_name:tag> /bin/bash",
                "summary": "AdoptOpenJDK Docker Image for OpenJDK with hotspot and ubi",
                "url": "https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.2-347",
                "vcs-ref": "663db861f0ff7a9c526c1c169a62c14c01a32dcc",
                "vcs-type": "git",
                "vendor": "AdoptOpenJDK",
                "version": "jdk-11.0.8+10"
            }
        },
        "NetworkSettings": {
            "Bridge": "",
            "SandboxID": "ec932b5d230a7b75445b794e756a19f175470dcbdcc9d8c9adc5333e9c61faa1",
            "HairpinMode": false,
            "LinkLocalIPv6Address": "",
            "LinkLocalIPv6PrefixLen": 0,
            "Ports": {
                "9000/tcp": [
                    {
                        "HostIp": "0.0.0.0",
                        "HostPort": "9000"
                    },
                    {
                        "HostIp": "::",
                        "HostPort": "9000"
                    }
                ]
            },
            "SandboxKey": "/var/run/docker/netns/ec932b5d230a",
            "SecondaryIPAddresses": null,
            "SecondaryIPv6Addresses": null,
            "EndpointID": "",
            "Gateway": "",
            "GlobalIPv6Address": "",
            "GlobalIPv6PrefixLen": 0,
            "IPAddress": "",
            "IPPrefixLen": 0,
            "IPv6Gateway": "",
            "MacAddress": "",
            "Networks": {
                "akkaserverless-geo-fence_default": {
                    "IPAMConfig": null,
                    "Links": null,
                    "Aliases": [
                        "akka-serverless-proxy",
                        "9df066fd8856"
                    ],
                    "NetworkID": "d7210bb12de632b9189c4f398974e0b7a5a27e702adcfdaa0a50e21ba4f1a292",
                    "EndpointID": "5a00ed98449b87bf54995c457107a02e232b94e29933b718c8bac2027002b0a1",
                    "Gateway": "192.168.0.1",
                    "IPAddress": "192.168.0.2",
                    "IPPrefixLen": 20,
                    "IPv6Gateway": "",
                    "GlobalIPv6Address": "",
                    "GlobalIPv6PrefixLen": 0,
                    "MacAddress": "02:42:c0:a8:00:02",
                    "DriverOpts": null
                }
            }
        }
    }
]

Of interest, I see that there is no IP for the port binding:

            "NetworkMode": "akkaserverless-geo-fence_default",
            "PortBindings": {
                "9000/tcp": [
                    {
                        "HostIp": "",
                        "HostPort": "9000"
                    }
                ]
            },

and later, I do see the network binding:

        "NetworkSettings": {
            "Bridge": "",
...
            "Ports": {
                "9000/tcp": [
                    {
                        "HostIp": "0.0.0.0",
                        "HostPort": "9000"
                    },
                    {
                        "HostIp": "::",
                        "HostPort": "9000"
                    }
                ]
            },

Hi @Michael-Read, we’re you able to resolve your issue or are you still running into it?

no luck. I can see the bridge, and can ping both the proxy and pubsub emulator directly on the docker bridge, however, I just can’t access the proxy on port 9000. I’m beginning to believe the networking part is fine, but the proxy has a problem. This morning I see these two logs repeating on docker-compose:

akka-serverless-proxy_1   | {"timestamp":"2021-07-02T15:45:56.149Z","thread":"akkaserverless-proxy-akka.actor.default-dispatcher-14","mdc":{"akkaAddress":"akka://akkaserverless-proxy@172.21.0.2:25520","sourceThread":"akkaserverless-proxy-akka.actor.default-dispatcher-5","akkaSource":"akka://akkaserverless-proxy/user/server-manager-supervisor/server-manager","sourceActorSystem":"akkaserverless-proxy","akkaTimestamp":"15:45:56.148UTC"},"logger":"akka.actor.OneForOneStrategy","message":"UNAVAILABLE: io exception","context":"default","exception":"io.grpc.StatusRuntimeException: UNAVAILABLE: io exception\n\tat io.grpc.Status.asRuntimeException(Status.java:534)\n\tat akka.grpc.internal.UnaryCallAdapter.onClose(UnaryCallAdapter.scala:40)\n\tat io.grpc.internal.DelayedClientCall$DelayedListener$3.run(DelayedClientCall.java:464)\n\tat io.grpc.internal.DelayedClientCall$DelayedListener.delayOrExecute(DelayedClientCall.java:428)\n\tat io.grpc.internal.DelayedClientCall$DelayedListener.onClose(DelayedClientCall.java:461)\n\tat io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:553)\n\tat io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:68)\n\tat io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInternal(ClientCallImpl.java:739)\n\tat io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:718)\n\tat io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)\n\tat io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)\n\tat java.base/java.lang.Thread.run(Unknown Source)\nCaused by: io.grpc.netty.shaded.io.netty.channel.ConnectTimeoutException: connection timed out: host.docker.internal/172.17.0.1:8080\n\tat io.grpc.netty.shaded.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe$2.run(AbstractEpollChannel.java:575)\n\tat io.grpc.netty.shaded.io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98)\n\tat io.grpc.netty.shaded.io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:170)\n\tat io.grpc.netty.shaded.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164)\n\tat io.grpc.netty.shaded.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472)\n\tat io.grpc.netty.shaded.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384)\n\tat io.grpc.netty.shaded.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)\n\tat io.grpc.netty.shaded.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)\n\tat io.grpc.netty.shaded.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)\n\t... 1 common frames omitted\n","severity":"ERROR"}
akka-serverless-proxy_1   | {"timestamp":"2021-07-02T15:45:57.321Z","thread":"akkaserverless-proxy-akka.actor.default-dispatcher-5","mdc":{"akkaAddress":"akka://akkaserverless-proxy@172.21.0.2:25520","sourceThread":"akkaserverless-proxy-akka.actor.default-dispatcher-11","akkaSource":"akka://akkaserverless-proxy@172.21.0.2:25520/user/server-manager-supervisor/server-manager","sourceActorSystem":"akkaserverless-proxy","akkaTimestamp":"15:45:57.320UTC"},"logger":"com.akkaserverless.proxy.DiscoveryManager","message":"Connecting to host.docker.internal:8080 to discover the user function specification","context":"default","severity":"INFO"}

The first is an ERROR, and the second one is an INFO.

The most important part of that ERROR log line is:

so it looks like there might still be something wrong with the networking.

I’ve run into something similar, but I resorted to the rather blunt iptables -I INPUT -i the_interface -j ACCEPT for the interface docker-compose was using.

Another thing to double-check is making sure that your user function is binding to 0.0.0.0, I had to do that explicitly as well. You can check with ‘netstat -pan’ on your host machine.

hmmm, that 172.17.0.1 is on the docker0 network, not the same bridge of the proxy, so I suppose that’s the host.docker.internal. I’m not sure how to identify the proper interface for the blunt force approach. If the bridge for the proxy, and docker0 need to route traffic, wouldn’t the docker-compose require both “networks”? I’ll play around with that some more instead of relying on the extra-host,

I see this from 'netstat -pan` whether my function is running or not:

tcp        0      0 0.0.0.0:9000            0.0.0.0:*               LISTEN      -       

I’m not sure what to look for to identify my user function. I greped for java but wasn’t able to decern anything.

okay, I’m starting to understand the plumbing… so I find this when I don’t have docker-compose running and, then run the function locally with mvn compile exec:java, and then grep 8080:

tcp6       0      0 127.0.0.1:8080          :::*                    LISTEN      32176/java      

Is tcp6 correct?, but it bound to 127.0.0.1 not 0.0.0.0

With this running I can actually do a

curl localhost:8080/

no error.

But if not running, then curl: (7) Failed to connect to localhost port 8080: Connection refused

I haven’t figured out how to bind to 0.0.0.0 with mvn exec:java

One last comment before giving up for the long weekend.

export USER_FUNCTION_HOST=localhost
michael@hpspectre:~/Downloads/Projects/Hackathon/akkaserverless-geo-fence$ docker-compose -f docker-compose.yml -f docker-compose.linux.yml up

causes this error

Connection refused: localhost/127.0.0.1:8080

You can list all networks with docker network ls. Look for the network where the NAME is foo_default, where foo is the name of the directory where docker-compose was invoked.

Then you take the value in the NETWORK ID column (e.g. 3aa3ca34880a) and prepend br-: br-3aa3ca34880a should be the interface.

Yes, that’s OK

Yeah, that’s currently the default - you can override that by setting the HOST environment variable (akkaserverless-java-sdk/reference.conf at 03325554ab7b36586340777ac475627db6e371f0 · lightbend/akkaserverless-java-sdk · GitHub)

Thank you Arnout for the weekend response. I’ll give it a try after the long weekend. :fireworks:

1 Like

Oh, so HOST must be provided through the environment?

Okay, so I updated the docker-compose.yaml to look like this:

version: "3"
services:
  akka-serverless-proxy:
    image: gcr.io/akkaserverless-public/akkaserverless-proxy:0.7.0-beta.9
    command: -Dconfig.resource=dev-mode.conf -Dakkaserverless.proxy.eventing.support=google-pubsub-emulator
    ports:
      - "9000:9000"
    environment:
#      USER_FUNCTION_HOST: ${USER_FUNCTION_HOST:-host.docker.internal}
#      USER_FUNCTION_PORT: ${USER_FUNCTION_PORT:-8080}
      HOST: 0.0.0.0
  gcloud-pubsub-emulator:
    image: gcr.io/google.com/cloudsdktool/cloud-sdk:341.0.0
    command: gcloud beta emulators pubsub start --project=test --host-port=0.0.0.0:8085
    ports:
      - 8085:8085

I also tried leaving the USER_FUNCTION_HOST, and PORT, in both cases it doesn’t seem to pick up the HOST environment variable. The error still looks like this:

{
  "timestamp": "2021-07-06T17:26:33.332Z",
  "thread": "akkaserverless-proxy-akka.actor.default-dispatcher-2",
  "mdc": {
    "akkaAddress": "akka://akkaserverless-proxy@192.168.0.2:25520",
    "sourceThread": "akkaserverless-proxy-akka.actor.default-dispatcher-3",
    "akkaSource": "akka://akkaserverless-proxy/user/server-manager-supervisor/server-manager",
    "sourceActorSystem": "akkaserverless-proxy",
    "akkaTimestamp": "17:26:33.330UTC"
  },
  "logger": "akka.actor.OneForOneStrategy",
  "message": "UNAVAILABLE: io exception",
  "context": "default",
  "exception": "io.grpc.StatusRuntimeException: UNAVAILABLE: io exception\n\tat io.grpc.Status.asRuntimeException(Status.java:534)\n\tat akka.grpc.internal.UnaryCallAdapter.onClose(UnaryCallAdapter.scala:40)\n\tat io.grpc.internal.DelayedClientCall$DelayedListener$3.run(DelayedClientCall.java:464)\n\tat io.grpc.internal.DelayedClientCall$DelayedListener.delayOrExecute(DelayedClientCall.java:428)\n\tat io.grpc.internal.DelayedClientCall$DelayedListener.onClose(DelayedClientCall.java:461)\n\tat io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:553)\n\tat io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:68)\n\tat io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInternal(ClientCallImpl.java:739)\n\tat io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:718)\n\tat io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)\n\tat io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)\n\tat java.base/java.lang.Thread.run(Unknown Source)\nCaused by: io.grpc.netty.shaded.io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: Connection refused: /127.0.0.1:8080\nCaused by: java.net.ConnectException: finishConnect(..) failed: Connection refused\n\tat io.grpc.netty.shaded.io.netty.channel.unix.Errors.throwConnectException(Errors.java:124)\n\tat io.grpc.netty.shaded.io.netty.channel.unix.Socket.finishConnect(Socket.java:251)\n\tat io.grpc.netty.shaded.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.doFinishConnect(AbstractEpollChannel.java:672)\n\tat io.grpc.netty.shaded.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:649)\n\tat io.grpc.netty.shaded.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:529)\n\tat io.grpc.netty.shaded.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:465)\n\tat io.grpc.netty.shaded.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378)\n\tat io.grpc.netty.shaded.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)\n\tat io.grpc.netty.shaded.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)\n\tat io.grpc.netty.shaded.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)\n\tat java.base/java.lang.Thread.run(Unknown Source)\n",
  "severity": "ERROR"
}

This is adding the HOST environment variable to the environment of the proxy - you want to add it to the environment of your application itself, so HOST=0.0.0.0 mvn compile exec:java. After that you should be able to verify with netstat that your application is listening on 0.0.0.0. It would be interesting to see whether the proxy can then connect to it - if not then there’s likely still some routing/firewalling that needs to be tweaked…