[akka-http] The http server closed the connection unexpectedly before delivering responses for 1 outstanding requests

Akka Akka HTTP
1

I have 2 containers, using docker command --net host , the akka application in a container, makes request to another container(a file server using nginx) .

The error information is:

akka.http.impl.engine.client.OutgoingConnectionBlueprint$.$anonfun$apply$6(OutgoingConnectionBlueprint.scala:123)
akka.http.impl.engine.client.OutgoingConnectionBlueprint$.$anonfun$apply$6$adapted(OutgoingConnectionBlueprint.scala:123)
akka.http.impl.util.One2OneBidiFlow$One2OneBidi$$anon$1$$anon$4.onUpstreamFinish(One2OneBidiFlow.scala:97)
akka.stream.impl.fusing.GraphInterpreter.processEvent(GraphInterpreter.scala:523)
akka.stream.impl.fusing.GraphInterpreter.execute(GraphInterpreter.scala:390)
akka.stream.impl.fusing.GraphInterpreterShell.runBatch(ActorGraphInterpreter.scala:625)
akka.stream.impl.fusing.GraphInterpreterShell$AsyncInput.execute(ActorGraphInterpreter.scala:502)
akka.stream.impl.fusing.GraphInterpreterShell.processEvent(ActorGraphInterpreter.scala:600)
akka.stream.impl.fusing.ActorGraphInterpreter.akka$stream$impl$fusing$ActorGraphInterpreter$$processEvent(ActorGraphInterpreter.scala:769)
akka.stream.impl.fusing.ActorGraphInterpreter$$anonfun$receive$1.applyOrElse(ActorGraphInterpreter.scala:784)
akka.actor.Actor.aroundReceive(Actor.scala:535)
akka.actor.Actor.aroundReceive$(Actor.scala:533)
akka.stream.impl.fusing.ActorGraphInterpreter.aroundReceive(ActorGraphInterpreter.scala:691)
akka.actor.ActorCell.receiveMessage(ActorCell.scala:575)
akka.actor.ActorCell.invoke(ActorCell.scala:545)
akka.dispatch.Mailbox.processMailbox(Mailbox.scala:270)
akka.dispatch.Mailbox.run(Mailbox.scala:231)
akka.dispatch.Mailbox.exec(Mailbox.scala:243)
java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)
java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056)
java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692)
java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:157)

akkaVersion = "2.6.5"
akkaHttpVersion = "10.1.12"

config:

akka {
    http {
      host-connection-pool {
        max-connections = 1024
        min-connections = 1024
        max-open-requests = 16384
      }
    }
  }
2

Hi @mggger,

there’s not much more to say than the error message: the server closed the connection before it sent a response. The best guess is that it’s a problem on the server side.

Johannes

3

Hi @jrudolph
We are also receiving this error at high rate recently. Any idea how to debug such error? I would like to know more what causes it. I don’t mind changing Akka HTTP code (adding some log messages) and trying it out.
Requests are send through proxy (ProxyMesh).

I’m attaching logs with error, maybe it will help you somehow:

DEBUG a.i.TcpOutgoingConnection           - Resolving us-dc.proxymesh.com before connecting
DEBUG a.i.SimpleDnsManager                - Resolution request for us-dc.proxymesh.com from Actor[akka://bellhop-service-api/system/IO-TCP/selectors/$a/64#1368963250]
DEBUG a.i.TcpOutgoingConnection           - Attempting connection to [us-dc.proxymesh.com/162.213.37.117:31280]
DEBUG a.h.i.e.c.PoolGateway               - Dispatching request [POST /v1/users Strict(284 bytes)] to pool
DEBUG a.i.TcpOutgoingConnection           - Attempting connection to [us-dc.proxymesh.com/162.213.37.117:31280]
DEBUG a.h.i.e.c.PoolGateway               - [2 (Idle)] Dispatching request [POST /v1/users Strict(284 bytes)]
DEBUG a.h.i.e.c.PoolGateway               - [2 (Idle)] Before event [onNewRequest] In state [Idle] for [20378 ms]
DEBUG a.h.i.e.c.PoolGateway               - [2 (PushingRequestToConnection)] After event [onNewRequest] State change [Idle] -> [PushingRequestToConnection]
DEBUG a.h.i.e.c.PoolGateway               - [2 (PushingRequestToConnection)] Before event [onRequestDispatched] In state [PushingRequestToConnection] for [0 ms]
DEBUG a.h.i.e.c.PoolGateway               - [2 (WaitingForResponse)] After event [onRequestDispatched] State change [PushingRequestToConnection] -> [WaitingForResponse]
DEBUG a.h.i.e.c.PoolGateway               - [2 (WaitingForResponse)] Connection failed
DEBUG a.h.i.e.c.PoolGateway               - [2 (WaitingForResponse)] Before event [onConnectionFailed] In state [WaitingForResponse] for [1 ms]
DEBUG a.h.i.e.c.PoolGateway               - [2 (WaitingForResponse)] Ongoing request [POST /v1/users Strict(284 bytes)] is failed because of [connection failure]: [The http server closed the connection unexpectedly before delivering responses for 1 outstanding requests]
DEBUG a.h.i.e.c.PoolGateway               - [2 (WaitingForResponseDispatch)] After event [onConnectionFailed] State change [WaitingForResponse] -> [WaitingForResponseDispatch]
DEBUG a.h.i.e.c.PoolGateway               - [2 (WaitingForResponseDispatch)] Before event [onResponseDispatchable] In state [WaitingForResponseDispatch] for [0 ms]
DEBUG a.h.i.e.c.PoolGateway               - [2 (Failed)] After event [onResponseDispatchable] State change [WaitingForResponseDispatch] -> [Failed]
DEBUG a.h.i.e.c.PoolGateway               - [2 (Failed)] State change from [WaitingForResponseDispatch] to [Failed(akka.http.impl.engine.client.OutgoingConnectionBlueprint$UnexpectedConnectionClosureException: The http server closed the connection unexpectedly before delivering responses for 1 outstanding requests)]. Closing the existing connection.
DEBUG a.h.i.e.c.PoolGateway               - [2 (Unconnected)] Slot became idle... Trying to pull

Any pointers are much anticipated.

4

Again, not much more to say then what the error says: the server failed the connection before sending a response. You will have to check the server-side (or proxy) and find out what is happening over there.