Lagom 1.6.1 + Play 2.8 vs Lagom 1.5.5 + Play 2.7.4

I created a reproducible problem combining two templates:

  • lagom/lagom-scala.g8
  • playframework/play-scala-seed.g8

This is loosely based on this helpful article:

The difference is that I used Scala only and compile-time dependency injection.
I made two variants of this project:

  • Branch lagom-1.6.1 is based on Lagom 1.6.1 + Play 2.8 compiled with Scala 2.13.1
  • Branch lagom-1.5.5 is based on Lagom 1.5.5 + Play 2.7.4 compiled with Scala 2.12.9

With Lagom 1.5.5, the service gateway works as expected; that is,
it responds to http://localhost:9000/* requests.

With Lagom 1.6.1, the service gateway doesn’t work at all.
Requests to http://localhost:9000/* are rejected with an error that suggests
that the service gateway would be listening on http://0.0.0.0:9000
according to the error I see in sbt:

2020-04-13T03:37:11.253Z [error] akka.actor.ActorSystemImpl [akkaAddress=akka://application, sourceThread=application-akka.actor.default-dispatcher-18, akkaSource=akka.actor.ActorSystemImpl(application), sourceActorSystem=application, akkaTimestamp=03:37:11.245UTC] - Internal server error, sending 500 response
akka.stream.StreamTcpException: Tcp command [Connect(0.0.0.0:9000,None,List(),Some(10 seconds),true)] failed because of java.net.ConnectException: Connection refused
Caused by: java.net.ConnectException: Connection refused
        at java.base/sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
        at java.base/sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:779)
        at akka.io.TcpOutgoingConnection$$anonfun$connecting$1.$anonfun$applyOrElse$3(TcpOutgoingConnection.scala:104)
        at akka.io.TcpOutgoingConnection.akka$io$TcpOutgoingConnection$$reportConnectFailure(TcpOutgoingConnection.scala:51)
        at akka.io.TcpOutgoingConnection$$anonfun$connecting$1.applyOrElse(TcpOutgoingConnection.scala:104)
        at akka.actor.Actor.aroundReceive(Actor.scala:534)
        at akka.actor.Actor.aroundReceive$(Actor.scala:532)
        at akka.io.TcpConnection.aroundReceive(TcpConnection.scala:32)
        at akka.actor.ActorCell.receiveMessage(ActorCell.scala:573)
  | => Tat akka.actor.ActorCell.invoke(ActorCell.scala:543)
        at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:269)
        at akka.dispatch.Mailbox.run(Mailbox.scala:230)
        at akka.dispatch.Mailbox.exec(Mailbox.scala:242)
        at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:290)
        at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1020)
        at java.base/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1656)
        at java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1594)
        at java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:177)

Yet, if I start with: sbt -Dhttp.address=127.0.0.1 runAll, then requests
to http://localhost:9000/* just hang.

All the details are in the README.md of both branches.

Does anyone know what’s going on w/ the Lagom 1.6 ServiceGateway?

  • Nicolas.

Hi @NicolasRouquette,

the frontend project is not properly setup:

 {
    "name": "frontend",
    "url": "http://0.0.0.0:9000",
    "portName": "http"
  },
  {
    "name": "frontend",
    "url": "http://0.0.0.0:9000",
    "portName": null
  },

The above indicates that the frontend project is registered in the service registry as being available in port 9000. That’s not true according to the sbt runAll output:

[info] Service hello-world-impl listening for HTTP on 127.0.0.1:53713
[info] Service frontend listening for HTTP on 127.0.0.1:60417
[info] Service hello-world-stream-impl listening for HTTP on 127.0.0.1:49636

As a consequence, hitting 127.0.0.1:9000 causes an infinite loop because the gateway thinks the frontend is on port 9000 and redirects the request to that port (causing the infinite loop).

Can you verify that frontend was properly registered in Lagom 1.5.x? What is the output for localhost:9008/services when running Lagom 1.5.x?

I’m not sure why that’s happening but looks like a regression. In the meantime, have you tried setting the port of frontedn using lagomServiceHttpsPort := 20443? (https://www.lagomframework.com/documentation/current/scala/ConfiguringServicesInDevelopment.html#How-are-ports-assigned-to-services?)

Cheers,

Indeed, I’ve been perplexed by this discrepancy between the logs from lagom 1.6.1 vs. 1.5.5.
The behavior w/ Lagom 1.5.5 looks completely normal, see:

The behavior w/ Lagom 1.6.1 is puzzling.

I managed to confirm that the service gateway is created w/ the correct address: http://127.0.0.1:9000
I can see this with a breakpoint in AkkaHttpServiceGateway.address

The puzzling part comes from the registration which is completely different!
With a breakpoint in ServiceRegistryImpl.register, I see that the Akka message is:

PUT /services/frontend HTTP/1.1
User-Agent: frontend
Content-Type: application/json
content-length: 69
host: 127.0.0.1:9008
accept: */*

{"uris":["http://0.0.0.0:9000"],"acls":[{"pathRegex":"(?!/api/).*"}]}

I am trying to figure out where the registration request comes from.
It’s tricky to put a breakpoint on that because ServiceRegistryImpl.register is
called in handling the request, which happens on a different thread than
the thread where the request is sent.

  • Nicolas.

The problem seems to be independent of the lagomServiceGatewayImpl; that is, netty and akka-http produce the same behavior in lagom 1.6.1

With the debugger, I managed to track down the approximate cause of the problem in Lagom’s ServiceRouter.handleServiceCall

  private def handleServiceCall[Request, Response](
      serviceCall: ServiceCall[Request, Response],
      descriptor: Descriptor,
      requestSerializer: MessageSerializer[Request, ByteString],
      responseSerializer: MessageSerializer[Response, ByteString],
      requestHeader: RequestHeader,
      playRequestHeader: PlayRequestHeader
  ): Accumulator[ByteString, Result] = {
    val requestMessageDeserializer =
      messageSerializerDeserializer(requestSerializer, messageHeaderProtocol(requestHeader))

    // Buffer the body in memory
    inMemoryBodyParser(playRequestHeader).mapFuture {
      // Error handling.
      // If it's left of a result (which this particular body parser should never return) just return that result.
      case Left(result) => Future.successful(result)
      // If the payload was too large, throw that exception (exception serializer will handle it later).
      case Right(Left(_)) =>
        throw newPayloadTooLarge("Request body larger than " + httpConfiguration.parser.maxMemoryBuffer)
      // Body was successfully buffered.
      case Right(Right(body)) =>
        // Deserialize request
        val request = negotiatedDeserializerDeserialize(requestMessageDeserializer, body)

        // Invoke the service call
        invokeServiceCall(serviceCall, requestHeader, request).map { ...

In the above, I see:

body =

PUT /services/frontend HTTP/1.1
User-Agent: frontend
Content-Type: application/json
content-length: 69
host: 127.0.0.1:9008
accept: */*

{"uris":["http://0.0.0.0:9000"],"acls":[{"pathRegex":"(?!/api/).*"}]}

requestHeader =

RequestHeader{method=PUT, uri=/services/frontend, protocol=MessageProtocol{contentType=Optional[application/json], charset=Optional.empty, version=Optional.empty}, acceptedResponseProtocols=[MessageProtocol{contentType=Optional[*/*], charset=Optional.empty, version=Optional.empty}], principal=Optional[com.lightbend.lagom.javadsl.api.security.ServicePrincipal$$Lambda$18549/0x0000000845372c40@3a6d3e5f], headers={User-Agent=[frontend], Remote-Address=[127.0.0.1:45724], Host=[127.0.0.1:9008], Tls-Session-Info=[Session(1586975041925|SSL_NULL_WITH_NULL_NULL)], Content-Type=[application/json], Raw-Request-URI=[/services/frontend], Content-Length=[69], Accept=[*/*], Timeout-Access=[<function1>]}}

Clearly, the playRequestHeader is suspicious (it is difficult to see the value in the debugger).
I hope that this is the correct avenue of investigation for the root cause of this problem.

  • Nicolas.

That trace is an indicator of the bug. The payload in this POST request is causing the issue:

PUT /services/frontend HTTP/1.1
User-Agent: frontend
Content-Type: application/json
content-length: 69
host: 127.0.0.1:9008
accept: */*

{"uris":["http://0.0.0.0:9000"],"acls":[{"pathRegex":"(?!/api/).*"}]}

frontend is not bound to "http://0.0.0.0:9000".

Would you raise an issue in Lagom?

Thanks!

Ok, I’m preparing an issue. - Nicolas.

Here’s the issue: https://github.com/lagom/lagom/issues/2823

1 Like