Cant start lago service in production mode - Oops, cannot start the server error

My lagom project works fine with command “mvn lagom:runAll”.

However, same fails with following error when run as standalone. I am basically creating an executable jar and trying to run that jar from commandline.

This is the script I am using for starting the service from command line.

#!/bin/sh

JVM_OPTS="-Xmx128m -Xms128m"

# This should be changed if you use Play sessions
PLAY_SECRET=none

# WARNING: lagom.cluster.join-self=on is not safe to use when running more than
# one instance of the service. For production scenarios you must configure
# clustering correctly for your deployment environment.
# See https://www.lagomframework.com/documentation/current/java/Cluster.html
CONFIG="-Dplay.crypto.secret=$PLAY_SECRET -Dlagom.cluster.join-self=on"

DIR=$(dirname $0)

java -cp ""./some-service-1.0.jar $JAVA_OPTS $CONFIG play.core.server.ProdServerStart

And this is the error I am getting.

Oops, cannot start the server.
com.google.inject.CreationException: Unable to create injector, see the following errors:

  1. No implementation for com.lightbend.lagom.javadsl.api.ServiceLocator was bound.
    while locating com.lightbend.lagom.javadsl.api.ServiceLocator
    for the 6th parameter of com.lightbend.lagom.internal.javadsl.broker.kafka.JavadslRegisterTopicProducers.(JavadslRegisterTopicProducers.scala:37)
    at com.lightbend.lagom.internal.javadsl.broker.kafka.KafkaBrokerModule.bindings(KafkaBrokerModule.scala:15):
    Binding(class com.lightbend.lagom.internal.javadsl.broker.kafka.JavadslRegisterTopicProducers to self eagerly) (via modules: com.google.inject.util.Modules$OverrideModule -> play.api.inject.guice.GuiceableModuleConversions$$anon$4)

  2. No implementation for com.lightbend.lagom.javadsl.api.ServiceLocator was bound.
    while locating com.lightbend.lagom.javadsl.api.ServiceLocator
    for the 4th parameter of com.lightbend.lagom.internal.javadsl.client.JavadslServiceClientImplementor.(JavadslServiceClientImplementor.scala:46)
    at com.lightbend.lagom.internal.javadsl.client.ServiceClientModule.bindings(ServiceClientModule.scala:15):
    Binding(class com.lightbend.lagom.internal.javadsl.client.JavadslServiceClientImplementor to self) (via modules: com.google.inject.util.Modules$OverrideModule -> play.api.inject.guice.GuiceableModuleConversions$$anon$4)

  3. No implementation for com.lightbend.lagom.javadsl.api.ServiceLocator was bound.
    while locating com.lightbend.lagom.javadsl.api.ServiceLocator
    for the 5th parameter of com.lightbend.lagom.javadsl.broker.kafka.KafkaTopicFactory.(KafkaTopicFactory.java:58)
    at play.inject.BindingKey.to(BindingKey.java:127):
    Binding(interface com.lightbend.lagom.internal.javadsl.api.broker.TopicFactory to ConstructionTarget(class com.lightbend.lagom.javadsl.broker.kafka.KafkaTopicFactory)) (via modules: com.google.inject.util.Modules$OverrideModule -> play.api.inject.guice.GuiceableModuleConversions$$anon$4)

3 errors
at com.google.inject.internal.Errors.throwCreationExceptionIfErrorsExist(Errors.java:543)
at com.google.inject.internal.InternalInjectorCreator.initializeStatically(InternalInjectorCreator.java:159)
at com.google.inject.internal.InternalInjectorCreator.build(InternalInjectorCreator.java:106)
at com.google.inject.Guice.createInjector(Guice.java:87)
at com.google.inject.Guice.createInjector(Guice.java:78)
at play.api.inject.guice.GuiceBuilder.injector(GuiceInjectorBuilder.scala:201)
at play.api.inject.guice.GuiceApplicationBuilder.build(GuiceApplicationBuilder.scala:156)
at play.api.inject.guice.GuiceApplicationLoader.load(GuiceApplicationLoader.scala:22)
at play.core.server.ProdServerStart$.start(ProdServerStart.scala:59)
at play.core.server.ProdServerStart$.main(ProdServerStart.scala:31)
at play.core.server.ProdServerStart.main(ProdServerStart.scala)

Thanks in advance
Naveena

Hi @Naveena,

This is happening because you didn’t configure a ServiceLocator for production mode. In DevMode Lagom configures a ServiceLocator for you, but in ProdMode you need to configure one because we can’t know in which environment you are going to deploy.

Please, read this section of the documentation.
https://www.lagomframework.com/documentation/current/java/ProductionOverview.html#Running-Lagom-in-production

Kind regards,

Renato

Thanks @renato. I am not seeing the error I mentioned before but seeing new error.

2019-10-12T13:51:16.274Z [warn] akka.remote.transport.netty.NettyTransport [] - Remote connection to [null] failed with java.net.ConnectException: Connection refused: no further information: /127.0.0.1:55706
2019-10-12T13:51:16.276Z [warn] akka.remote.ReliableDeliverySupervisor [] - Association with remote system [akka.tcp://uncompress-adc-service@127.0.0.1:55706] has failed, address is now gated for [5000] ms. Reason: [Association failed with [akka.tcp://uncompress-adc-service@127.0.0.1:55706]] Caused by: [java.net.ConnectException: Connection refused: no further information: /127.0.0.1:55706]

[Edit] Here are some more logs I thoughts might be useful.

[INFO] Service uncompress-adc-service listening for HTTP on 127.0.0.1:58365
[INFO] (Service started, press enter to stop and go back to the console…)
2019-10-12T14:08:58.103Z [warn] akka.cluster.Cluster(akka://uncompress-adc-service-application) [] - Cluster Node [akka.tcp://uncompress-adc-service-application@127.0.0.1:64929] - Couldn’t join seed nodes after [2] attempts, will try again. seed-nodes=[akka.tcp://uncompress-adc-service@localhost:55706]
2019-10-12T14:08:59.116Z [warn] akka.remote.transport.netty.NettyTransport [] - Remote connection to [null] failed with java.net.ConnectException: Connection refused: no further information: localhost/127.0.0.1:55706
2019-10-12T14:08:59.116Z [warn] akka.remote.ReliableDeliverySupervisor [] - Association with remote system [akka.tcp://uncompress-adc-service@localhost:55706] has failed, address is now gated for [5000] ms. Reason: [Association failed with [akka.tcp://uncompress-adc-service@localhost:55706]] Caused by: [java.net.ConnectException: Connection refused: no further information: localhost/127.0.0.1:55706]
2019-10-12T14:09:00.081Z [warn] akka.cluster.sharding.ShardRegion [] - kafkaProducer-move-adc-status-change: No coordinator found to register. Probably, no seed-nodes configured and manual cluster join not performed? Total [4] buffered messages.
2019-10-12T14:09:02.107Z [warn] akka.cluster.sharding.ShardRegion [] - kafkaProducer-move-adc-status-change: No coordinator found to register. Probably, no seed-nodes configured and manual cluster join not performed? Total [4] buffered messages.
2019-10-12T14:09:03.125Z [warn] akka.cluster.Cluster(akka://uncompress-adc-service-application) [] - Cluster Node [akka.tcp://uncompress-adc-service-application@127.0.0.1:64929] - Couldn’t join seed nodes after [3] attempts, will try again. seed-nodes=[akka.tcp://uncompress-adc-service@localhost:55706]

Any clue on this?

I have added following section in my application.conf.

akka {
	discovery {
	   method = akka-dns
	  }
  actor {
    provider = "cluster"
  }
  remote {
    netty.tcp {
      hostname = "127.0.0.1"
      port = 0
    }

    artery {
      # change this to enabled=on to use Artery instead of netty
      # see https://doc.akka.io/docs/akka/current/remoting-artery.html
      enabled = off
      transport = tcp
      canonical.hostname = "127.0.0.1"
      canonical.port = 0
    }
  }
  
   cluster {
    # Note - Artery uses akka:// addresses
    seed-nodes = [
      "akka.tcp://uncompress-adc-service@127.0.0.1:55706"]

    # auto downing is NOT safe for production deployments.
    # you may want to use it during development, read more about it in the docs.
    auto-down-unreachable-after = 10s
  }
}

I am binding ServiceLocator and ConfigurationServiceLocator as defined here.

bind(ServiceLocator.class).to(ConfigurationServiceLocator.class);

Lot of stuffs are new to me, trying with trial and error. Thanks a stack for all the help!
Naveena

Hi @Naveena,

Your remote port is set to 0 meaning that a random port will be selected. Then, your seed nodes is configured to connect to port 55706. I guess you are trying to form a single node cluster and making it join itself in prod. Is that correct?

If that’s the case, you should simply use 25520 which is the default port for Artery. Make sure you have different settings for dev and prod.

Cheers,

Renato

Hi @renato Thanks for your replies. I realized the mistake and corrected and I did exactly what you noted. So now the service runs in prod mode as single node cluster. I am yet to venture into multinode cluster. Hope I just have to update the seed nodes list and deploy the services in those nodes as well.

Thanks for all your help
Naveena

You are welcome.

When moving to production you may be interested in not having seed-nodes but to use Cluster Bootstrap instead with DNS for looking up the nodes IP. Check this documentation section for more details: https://www.lagomframework.com/documentation/current/scala/Cluster.html#Joining

Of course, if you have static IP address but also fixed number of nodes, seed-nodes is the easiest way of doing it.

Have fun!

Cheers,

Renato

Hello @renato

Thank you for your response. As the first step, I am using static configuration which is working now. Next will try the dynamic configuration. Thank you for the link, will use that as the starting point.

Thanks again :slightly_smiling_face:
Naveena

I tried to replicate the same settings in another service but unable to start the service. As soon as I start the service, I get the following exception

INFO: Slf4jLogger started
Oct 25, 2019 12:05:37 PM akka.event.slf4j.Slf4jLogger$$anonfun$receive$1 $anonfun$applyOrElse$3
INFO: Starting remoting
Oct 25, 2019 12:05:37 PM akka.event.slf4j.Slf4jLogger$$anonfun$receive$1 $anonfun$applyOrElse$1
SEVERE: failed to bind to /192.168.65.3:2552, shutting down Netty transport
Oct 25, 2019 12:05:37 PM akka.event.slf4j.Slf4jLogger$$anonfun$receive$1 $anonfun$applyOrElse$1
SEVERE: Remoting error: [Startup failed] [
akka.remote.RemoteTransportException: Startup failed
        at akka.remote.Remoting.akka$remote$Remoting$$notifyError(Remoting.scala:163)
        at akka.remote.Remoting.start(Remoting.scala:238)

My observations were:

  1. The netstat shows that the “192.168.65.3:2552” is being used. I am very certain that I havent started any application on that port. Its a default openjdk:8 based docker image and no additional applications were installed.
  2. In my apllication.conf, the seed node port I have configured is 2553 and the IP address is 127.0.0.1.
    I dont know why the log shows that 192.168.65.3:2552 is used ignoring what was defined in application.conf.

Any help on second point above is very much appreciated.

Thanks in advance
Naveena

Pls ignore this query. I had issues with my container. Fixed the problem now.

Thanks
Naveena