No acknowledgement received when trying to register to coordinator, messages buffered.

Hello:

I am new to Lagom/Akka and trying to learn while doing. I have an app with three services that I am trying to run Docker via docker-compose.

The leader node seems to start just fine, no glaring warnings. Both non-leader members do not. Log below:

2020-04-21T19:52:44.121Z [info] akka.event.slf4j.Slf4jLogger [] - Slf4jLogger started
2020-04-21T19:52:44.609Z [info] akka.remote.artery.tcp.ArteryTcpTransport [akkaAddress=akka://application@app-datasource:25520, sourceThread=main, akkaSource=ArteryTcpTransport(akka://application), sourceActorSystem=application, akkaTimestamp=19:52:44.608UTC] - Remoting started with transport [Artery tcp]; listening on address [akka://application@app-datasource:25520] with UID [8651354229357041999]
2020-04-21T19:52:44.641Z [info] akka.cluster.Cluster [akkaAddress=akka://application@app-datasource:25520, sourceThread=main, akkaSource=Cluster(akka://application), sourceActorSystem=application, akkaTimestamp=19:52:44.641UTC] - Cluster Node [akka://application@app-datasource:25520] - Starting up, Akka version [2.6.3] ...
2020-04-21T19:52:44.813Z [info] akka.cluster.Cluster [akkaAddress=akka://application@app-datasource:25520, sourceThread=main, akkaSource=Cluster(akka://application), sourceActorSystem=application, akkaTimestamp=19:52:44.813UTC] - Cluster Node [akka://application@app-datasource:25520] - Registered cluster JMX MBean [akka:type=Cluster]
2020-04-21T19:52:44.813Z [info] akka.cluster.Cluster [akkaAddress=akka://application@app-datasource:25520, sourceThread=main, akkaSource=Cluster(akka://application), sourceActorSystem=application, akkaTimestamp=19:52:44.813UTC] - Cluster Node [akka://application@app-datasource:25520] - Started up successfully
2020-04-21T19:52:44.872Z [info] akka.cluster.Cluster [akkaAddress=akka://application@app-datasource:25520, sourceThread=application-akka.actor.internal-dispatcher-4, akkaSource=Cluster(akka://application), sourceActorSystem=application, akkaTimestamp=19:52:44.872UTC] - Cluster Node [akka://application@app-datasource:25520] - No downing-provider-class configured, manual cluster downing required, see https://doc.akka.io/docs/akka/current/typed/cluster.html#downing
2020-04-21T19:52:46.192Z [info] akka.management.cluster.bootstrap.ClusterBootstrap [akkaAddress=akka://application@app-datasource:25520, sourceThread=main, akkaSource=ClusterBootstrap(akka://application), sourceActorSystem=application, akkaTimestamp=19:52:46.190UTC] - Bootstrap using `akka.discovery` method: akka-dns
2020-04-21T19:52:46.212Z [info] akka.io.DnsExt [akkaAddress=akka://application@app-datasource:25520, sourceThread=main, akkaSource=DnsExt(akka://application), sourceActorSystem=application, akkaTimestamp=19:52:46.212UTC] - Creating async dns resolver async-dns with manager name SD-DNS
2020-04-21T19:52:46.280Z [info] akka.management.internal.HealthChecksImpl [akkaAddress=akka://application@app-datasource:25520, sourceThread=main, akkaSource=HealthChecksImpl(akka://application), sourceActorSystem=application, akkaTimestamp=19:52:46.280UTC] - Loading readiness checks List(NamedHealthCheck(cluster-membership,akka.management.cluster.scaladsl.ClusterMembershipCheck))
2020-04-21T19:52:46.280Z [info] akka.management.internal.HealthChecksImpl [akkaAddress=akka://application@app-datasource:25520, sourceThread=main, akkaSource=HealthChecksImpl(akka://application), sourceActorSystem=application, akkaTimestamp=19:52:46.280UTC] - Loading liveness checks List()
2020-04-21T19:52:46.395Z [warn] akka.stream.Materializer [akkaAddress=akka://application@app-datasource:25520, sourceThread=application-akka.remote.default-remote-dispatcher-10, akkaSource=akka.stream.Log(akka://application/system/Materializers/StreamSupervisor-1), sourceActorSystem=application, akkaTimestamp=19:52:46.390UTC] - [outbound connection to [akka://application@app-process:25520], control stream] Upstream failed, cause: StreamTcpException: Tcp command [Connect(app-process:25520,None,List(),Some(5000 milliseconds),true)] failed because of java.net.ConnectException: Connection refused
2020-04-21T19:52:46.395Z [warn] akka.stream.Materializer [akkaAddress=akka://application@app-datasource:25520, sourceThread=application-akka.remote.default-remote-dispatcher-6, akkaSource=akka.stream.Log(akka://application/system/Materializers/StreamSupervisor-1), sourceActorSystem=application, akkaTimestamp=19:52:46.392UTC] - [outbound connection to [akka://application@app-process:25520], message stream] Upstream failed, cause: StreamTcpException: Tcp command [Connect(app-process:25520,None,List(),Some(5000 milliseconds),true)] failed because of java.net.ConnectException: Connection refused
2020-04-21T19:52:46.404Z [warn] akka.stream.Materializer [akkaAddress=akka://application@app-datasource:25520, sourceThread=application-akka.remote.default-remote-dispatcher-12, akkaSource=akka.stream.Log(akka://application/system/Materializers/StreamSupervisor-1), sourceActorSystem=application, akkaTimestamp=19:52:46.392UTC] - [outbound connection to [akka://application@test:25520], control stream] Upstream failed, cause: StreamTcpException: Tcp command [Connect(test:25520,None,List(),Some(5000 milliseconds),true)] failed because of java.net.ConnectException: Connection refused
2020-04-21T19:52:46.416Z [warn] akka.stream.Materializer [akkaAddress=akka://application@app-datasource:25520, sourceThread=application-akka.remote.default-remote-dispatcher-8, akkaSource=akka.stream.Log(akka://application/system/Materializers/StreamSupervisor-1), sourceActorSystem=application, akkaTimestamp=19:52:46.391UTC] - [outbound connection to [akka://application@test:25520], message stream] Upstream failed, cause: StreamTcpException: Tcp command [Connect(test:25520,None,List(),Some(5000 milliseconds),true)] failed because of java.net.ConnectException: Connection refused
2020-04-21T19:52:46.497Z [info] akka.management.scaladsl.AkkaManagement [akkaAddress=akka://application@app-datasource:25520, sourceThread=main, akkaSource=AkkaManagement(akka://application), sourceActorSystem=application, akkaTimestamp=19:52:46.496UTC] - Binding Akka Management (HTTP) endpoint to: app-datasource:8558
2020-04-21T19:52:46.586Z [info] akka.management.scaladsl.AkkaManagement [akkaAddress=akka://application@app-datasource:25520, sourceThread=main, akkaSource=AkkaManagement(akka://application), sourceActorSystem=application, akkaTimestamp=19:52:46.586UTC] - Including HTTP management routes for ClusterHttpManagementRouteProvider
2020-04-21T19:52:46.667Z [info] akka.management.scaladsl.AkkaManagement [akkaAddress=akka://application@app-datasource:25520, sourceThread=main, akkaSource=AkkaManagement(akka://application), sourceActorSystem=application, akkaTimestamp=19:52:46.665UTC] - Including HTTP management routes for ClusterBootstrap
2020-04-21T19:52:46.673Z [info] akka.management.cluster.bootstrap.ClusterBootstrap [akkaAddress=akka://application@app-datasource:25520, sourceThread=main, akkaSource=ClusterBootstrap(akka://application), sourceActorSystem=application, akkaTimestamp=19:52:46.673UTC] - Using self contact point address: http://app-datasource:8558
2020-04-21T19:52:46.698Z [info] akka.management.scaladsl.AkkaManagement [akkaAddress=akka://application@app-datasource:25520, sourceThread=main, akkaSource=AkkaManagement(akka://application), sourceActorSystem=application, akkaTimestamp=19:52:46.698UTC] - Including HTTP management routes for HealthCheckRoutes
2020-04-21T19:52:47.616Z [info] akka.cluster.Cluster [akkaAddress=akka://application@app-datasource:25520, sourceThread=application-akka.actor.internal-dispatcher-13, akkaSource=Cluster(akka://application), sourceActorSystem=application, akkaTimestamp=19:52:47.616UTC] - Cluster Node [akka://application@app-datasource:25520] - Received InitJoin message from [Actor[akka://application@app-cron:25520/system/cluster/core/daemon/firstSeedNodeProcess-1#97626252]], but this node is not initialized yet
2020-04-21T19:52:47.619Z [info] akka.cluster.Cluster [akkaAddress=akka://application@app-datasource:25520, sourceThread=application-akka.actor.internal-dispatcher-13, akkaSource=Cluster(akka://application), sourceActorSystem=application, akkaTimestamp=19:52:47.619UTC] - Cluster Node [akka://application@app-datasource:25520] - Received InitJoin message from [Actor[akka://application@app-cron:25520/system/cluster/core/daemon/firstSeedNodeProcess-1#97626252]], but this node is not initialized yet
2020-04-21T19:52:47.736Z [info] akka.management.scaladsl.AkkaManagement [akkaAddress=akka://application@app-datasource:25520, sourceThread=application-akka.actor.default-dispatcher-32, akkaSource=AkkaManagement(akka://application), sourceActorSystem=application, akkaTimestamp=19:52:47.736UTC] - Bound Akka Management (HTTP) endpoint to: app-datasource:8558
2020-04-21T19:52:48.800Z [info] akka.cluster.sharding.ShardRegion [akkaAddress=akka://application@app-datasource:25520, sourceThread=application-akka.actor.internal-dispatcher-22, akkaSource=akka://application@app-datasource:25520/system/sharding/kafkaProducer-app_datasource, sourceActorSystem=application, akkaTimestamp=19:52:48.800UTC] - kafkaProducer-app_datasource: Idle entities will be passivated after [2.000 min]
2020-04-21T19:52:48.808Z [info] akka.cluster.sharding.typed.scaladsl.ClusterSharding [akkaAddress=akka://application@app-datasource:25520, sourceThread=main, akkaSource=ClusterSharding(akka://application), sourceActorSystem=application, akkaTimestamp=19:52:48.808UTC] - Starting Shard Region [DataSource]...
2020-04-21T19:52:48.814Z [info] akka.cluster.sharding.ShardRegion [akkaAddress=akka://application@app-datasource:25520, sourceThread=application-akka.actor.internal-dispatcher-15, akkaSource=akka://application@app-datasource:25520/system/sharding/DataSource, sourceActorSystem=application, akkaTimestamp=19:52:48.814UTC] - DataSource: Idle entities will be passivated after [2.000 min]
2020-04-21T19:52:48.921Z [info] play.api.Play [] - Application started (Prod) (no global state)
2020-04-21T19:52:48.961Z [info] play.core.server.AkkaHttpServer [] - Listening for HTTP on /0.0.0.0:9000
2020-04-21T19:52:49.395Z [info] akka.cluster.Cluster [akkaAddress=akka://application@app-datasource:25520, sourceThread=application-akka.actor.internal-dispatcher-14, akkaSource=Cluster(akka://application), sourceActorSystem=application, akkaTimestamp=19:52:49.394UTC] - Cluster Node [akka://application@app-datasource:25520] - Received InitJoin message from [Actor[akka://application@app-process:25520/system/cluster/core/daemon/joinSeedNodeProcess-1#618986798]], but this node is not initialized yet
2020-04-21T19:52:49.414Z [info] akka.cluster.Cluster [akkaAddress=akka://application@app-datasource:25520, sourceThread=application-akka.actor.internal-dispatcher-15, akkaSource=Cluster(akka://application), sourceActorSystem=application, akkaTimestamp=19:52:49.414UTC] - Cluster Node [akka://application@app-datasource:25520] - Received InitJoin message from [Actor[akka://application@test:25520/system/cluster/core/daemon/joinSeedNodeProcess-1#724457852]], but this node is not initialized yet
2020-04-21T19:52:54.496Z [info] akka.cluster.Cluster [akkaAddress=akka://application@app-datasource:25520, sourceThread=application-akka.actor.internal-dispatcher-22, akkaSource=Cluster(akka://application), sourceActorSystem=application, akkaTimestamp=19:52:54.496UTC] - Cluster Node [akka://application@app-datasource:25520] - Received InitJoin message from [Actor[akka://application@app-process:25520/system/cluster/core/daemon/joinSeedNodeProcess-1#618986798]], but this node is not initialized yet
2020-04-21T19:52:54.503Z [info] akka.cluster.Cluster [akkaAddress=akka://application@app-datasource:25520, sourceThread=application-akka.actor.internal-dispatcher-21, akkaSource=Cluster(akka://application), sourceActorSystem=application, akkaTimestamp=19:52:54.502UTC] - Cluster Node [akka://application@app-datasource:25520] - Received InitJoin message from [Actor[akka://application@test:25520/system/cluster/core/daemon/joinSeedNodeProcess-1#724457852]], but this node is not initialized yet
2020-04-21T19:52:54.638Z [info] akka.cluster.Cluster [akkaAddress=akka://application@app-datasource:25520, sourceThread=application-akka.actor.internal-dispatcher-3, akkaSource=Cluster(akka://application), sourceActorSystem=application, akkaTimestamp=19:52:54.638UTC] - Cluster Node [akka://application@app-datasource:25520] - Received InitJoinAck message from [Actor[akka://application@app-cron:25520/system/cluster/core/daemon#193428041]] to [akka://application@app-datasource:25520]
2020-04-21T19:52:55.494Z [info] akka.cluster.Cluster [akkaAddress=akka://application@app-datasource:25520, sourceThread=application-akka.actor.internal-dispatcher-23, akkaSource=Cluster(akka://application), sourceActorSystem=application, akkaTimestamp=19:52:55.493UTC] - Cluster Node [akka://application@app-datasource:25520] - Welcome from [akka://application@app-cron:25520]
2020-04-21T19:52:55.814Z [info] akka.cluster.singleton.ClusterSingletonManager [akkaAddress=akka://application@app-datasource:25520, sourceThread=application-akka.actor.internal-dispatcher-22, akkaSource=akka://application@app-datasource:25520/system/sharding/DataSourceCoordinator, sourceActorSystem=application, akkaTimestamp=19:52:55.814UTC] - ClusterSingletonManager state change [Start -> Younger]
2020-04-21T19:52:55.816Z [info] akka.cluster.singleton.ClusterSingletonManager [akkaAddress=akka://application@app-datasource:25520, sourceThread=application-akka.actor.internal-dispatcher-39, akkaSource=akka://application@app-datasource:25520/user/$a, sourceActorSystem=application, akkaTimestamp=19:52:55.814UTC] - ClusterSingletonManager state change [Start -> Younger]
2020-04-21T19:52:55.819Z [info] akka.cluster.singleton.ClusterSingletonManager [akkaAddress=akka://application@app-datasource:25520, sourceThread=application-akka.actor.internal-dispatcher-23, akkaSource=akka://application@app-datasource:25520/user/cassandraOffsetStorePrepare-singleton, sourceActorSystem=application, akkaTimestamp=19:52:55.817UTC] - ClusterSingletonManager state change [Start -> Younger]
2020-04-21T19:52:55.819Z [info] akka.cluster.singleton.ClusterSingletonManager [akkaAddress=akka://application@app-datasource:25520, sourceThread=application-akka.actor.internal-dispatcher-4, akkaSource=akka://application@app-datasource:25520/system/sharding/kafkaProducer-app_datasourceCoordinator, sourceActorSystem=application, akkaTimestamp=19:52:55.818UTC] - ClusterSingletonManager state change [Start -> Younger]
2020-04-21T19:52:56.318Z [info] akka.cluster.singleton.ClusterSingletonProxy [akkaAddress=akka://application@app-datasource:25520, sourceThread=application-akka.actor.internal-dispatcher-23, akkaSource=akka://application@app-datasource:25520/user/cassandraOffsetStorePrepare-singletonProxy, sourceActorSystem=application, akkaTimestamp=19:52:56.317UTC] - Singleton identified at [akka://application@app-cron:25520/user/cassandraOffsetStorePrepare-singleton/singleton]
2020-04-21T19:52:58.893Z [warn] akka.cluster.sharding.ShardRegion [akkaAddress=akka://application@app-datasource:25520, sourceThread=application-akka.actor.internal-dispatcher-33, akkaSource=akka://application@app-datasource:25520/system/sharding/kafkaProducer-app_datasource, sourceActorSystem=application, akkaTimestamp=19:52:58.893UTC] - kafkaProducer-app_datasource: Trying to register to coordinator at [ActorSelection[Anchor(akka://application@app-cron:25520/), Path(/system/sharding/kafkaProducer-app_datasourceCoordinator/singleton/coordinator)]], but no acknowledgement. Total [1] buffered messages. [Coordinator [Member(address = akka://application@app-cron:25520, status = Up)] is reachable.]
2020-04-21T19:53:00.913Z [warn] akka.cluster.sharding.ShardRegion [akkaAddress=akka://application@app-datasource:25520, sourceThread=application-akka.actor.internal-dispatcher-14, akkaSource=akka://application@app-datasource:25520/system/sharding/kafkaProducer-app_datasource, sourceActorSystem=application, akkaTimestamp=19:53:00.912UTC] - kafkaProducer-app_datasource: Trying to register to coordinator at [ActorSelection[Anchor(akka://application@app-cron:25520/), Path(/system/sharding/kafkaProducer-app_datasourceCoordinator/singleton/coordinator)]], but no acknowledgement. Total [1] buffered messages. [Coordinator [Member(address = akka://application@app-cron:25520, status = Up)] is reachable.]
2020-04-21T19:53:02.933Z [warn] akka.cluster.sharding.ShardRegion [akkaAddress=akka://application@app-datasource:25520, sourceThread=application-akka.actor.internal-dispatcher-33, akkaSource=akka://application@app-datasource:25520/system/sharding/kafkaProducer-app_datasource, sourceActorSystem=application, akkaTimestamp=19:53:02.933UTC] - kafkaProducer-app_datasource: Trying to register to coordinator at [ActorSelection[Anchor(akka://application@app-cron:25520/), Path(/system/sharding/kafkaProducer-app_datasourceCoordinator/singleton/coordinator)]], but no acknowledgement. Total [1] buffered messages. [Coordinator [Member(address = akka://application@app-cron:25520, status = Up)] is reachable.]
2020-04-21T19:53:04.953Z [warn] akka.cluster.sharding.ShardRegion [akkaAddress=akka://application@app-datasource:25520, sourceThread=application-akka.actor.internal-dispatcher-13, akkaSource=akka://application@app-datasource:25520/system/sharding/kafkaProducer-app_datasource, sourceActorSystem=application, akkaTimestamp=19:53:04.952UTC] - kafkaProducer-app_datasource: Trying to register to coordinator at [ActorSelection[Anchor(akka://application@app-cron:25520/), Path(/system/sharding/kafkaProducer-app_datasourceCoordinator/singleton/coordinator)]], but no acknowledgement. Total [1] buffered messages. [Coordinator [Member(address = akka://application@app-cron:25520, status = Up)] is reachable.]

The configuration:

play.application.loader = com.example.app.datasource.impl.DataStoreLoader

app.cassandra.keyspace = app_datasource

cassandra-journal.keyspace = ${app.cassandra.keyspace}
cassandra-snapshot-store.keyspace = ${app.cassandra.keyspace}
lagom.persistence.read-side.cassandra.keyspace = ${app.cassandra.keyspace}

elastic.url="http://localhost:9200"

akka.actor {
  serialization-bindings {
    # commands won't use play-json but Akka's jackson support
    "com.example.app.datasource.impl.command.CommandSerializable" = jackson-json
  }
}


include classpath("application.conf")

play.http.secret.key=abcdefghi12345567890

pidfile.path=/home/app/service.pid

elastic.url="http://elasticsearch:9200"
elastic.url=${?ELASTIC_URL}


cassandra.default {

  username = "cassandra"
  username = ${?CASSANDRA_USERNAME}

  password = "cassandra"
  password = ${?CASSANDRA_PASSWORD}

  ## list the contact points  here
  contact-points = ["cassandra"]

  ## override Lagom’s ServiceLocator-based ConfigSessionProvider
  session-provider = akka.persistence.cassandra.ConfigSessionProvider
}

cassandra-journal {
  contact-points = ${cassandra.default.contact-points}
  session-provider = ${cassandra.default.session-provider}
  authentication {
    username = ${cassandra.default.username}
    password = ${cassandra.default.password}
  }
}

cassandra-snapshot-store {
  contact-points = ${cassandra.default.contact-points}
  session-provider = ${cassandra.default.session-provider}
  authentication {
    username = ${cassandra.default.username}
    password = ${cassandra.default.password}
  }
}

lagom.persistence.read-side.cassandra {
  contact-points = ${cassandra.default.contact-points}
  session-provider = ${cassandra.default.session-provider}
  authentication {
    username = ${cassandra.default.username}
    password = ${cassandra.default.password}
  }
}

lagom.services {
  app-cron = "http://app-cron:9000"
  app-datasource = "http://app-datasource:9000"
  app-process = "http://app-process:9000"
  test = "http://test:9000"
}

akka.management.http {
  bind-hostname=${?AKKA_NAME}
  hostname=${?AKKA_NAME}
  port=8558
}

akka.cluster.seed-nodes = [
  "akka://application@app-cron:25520",
  "akka://application@app-datasource:25520",
  "akka://application@app-process:25520",
  "akka://application@test:25520"
]

akka {
  actor {
    # provider=remote is possible, but prefer cluster
    provider = cluster
  }
  remote {
    artery {
      transport = tcp # See Selecting a transport below
      canonical.hostname = ${?AKKA_NAME}
      canonical.port = 25520
    }
  }
}

akka.management {
  cluster.bootstrap {
    contact-point-discovery {
      discovery-method = akka-dns
      required-contact-point-nr = 1
    }
  }
}

akka {
  discovery {
    //method = akka-dns
    method = aggregate
    aggregate {
      discovery-methods = ["akka-dns", "config"]
    }
    config {
      services {
        kafka {
          endpoints = [
            {
              host = "kafka"
              port = 9092
            }
          ]
        }
        cassandra {
          endpoints = [
            {
              host = "cassandra"
              port = 9042
            }
          ]
        }
        app-cron {
          endpoints = [
            {
              host = "app-cron"
              port = 25520
            }
          ]
        }
        app-datasource {
          endpoints = [
            {
              host = "app-datasource"
              port = 25520
            }
          ]
        }
        app-process {
          endpoints = [
            {
              host = "app-process"
              port = 25520
            }
          ]
        }
        test {
          endpoints = [
            {
              host = "test"
              port = 25520
            }
          ]
        }
      }
    }
  }
}

I don’t quite understand what is going on where and don’t know where to go in solving it. Can someone offer suggestions??

I appreicate any help that is offered.
Thank you!!!

1 Like