Akka http2 upgrade does not work, throws IllegalArgumentException "Cannot pull port twice"

Hello,
in our akka.http server, I try to enable http2 support with config akka.http.server.preview.enable-http2 = on.
I test the connection upgrade with curl by sending

curl.exe -v http://localhost:9831 --http2
*   Trying 127.0.0.1:9831...
* Connected to localhost (127.0.0.1) port 9831 (#0)
> GET / HTTP/1.1
> Host: localhost:9831
> User-Agent: curl/7.80.0
> Accept: */*
> Connection: Upgrade, HTTP2-Settings
> Upgrade: h2c
> HTTP2-Settings: AAMAAABkAAQCAAAAAAIAAAAA
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 101 Switching Protocols
< Upgrade: h2c
< Server: NONE-LOCAL-1.0-1.0
< Date: Tue, 21 Dec 2021 10:09:12 GMT
< Connection: upgrade
* Received 101
* Using HTTP2, server supports multiplexing
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Send failure: Connection was aborted
* Failed sending HTTP2 data
* Send failure: Connection was aborted
* Failed sending HTTP2 data
* Connection #0 to host localhost left intact

In my akka app, I get Exception

2021-12-21 11:09:12,474 ERROR [a.a.RepointableActorRef::$anonfun$applyOrElse$1] Error during preStart in [akka.http.impl.engine.http2.Http2Demux$$anon$1]: Cannot pull port (Demux.substreamIn(621039725)) twice
java.lang.IllegalArgumentException: Cannot pull port (Demux.substreamIn(621039725)) twice
        at akka.stream.stage.GraphStageLogic.pull(GraphStage.scala:505)
        at akka.http.impl.engine.http2.Http2Demux$$anon$1.preStart(Http2Demux.scala:261)
        at akka.stream.impl.fusing.GraphInterpreter.init(GraphInterpreter.scala:306)
        at akka.stream.impl.fusing.GraphInterpreterShell.init(ActorGraphInterpreter.scala:594)
        at akka.stream.impl.fusing.ActorGraphInterpreter.tryInit(ActorGraphInterpreter.scala:702)
        at akka.stream.impl.fusing.ActorGraphInterpreter.finishShellRegistration(ActorGraphInterpreter.scala:745)
        at akka.stream.impl.fusing.ActorGraphInterpreter.akka$stream$impl$fusing$ActorGraphInterpreter$$shortCircuitBatch(ActorGraphInterpreter.scala:763)
        at akka.stream.impl.fusing.ActorGraphInterpreter$$anonfun$receive$1.applyOrElse(ActorGraphInterpreter.scala:789)
        at akka.actor.Actor.aroundReceive(Actor.scala:537)
        at akka.actor.Actor.aroundReceive$(Actor.scala:535)
        at akka.stream.impl.fusing.ActorGraphInterpreter.aroundReceive(ActorGraphInterpreter.scala:691)
        at akka.actor.ActorCell.receiveMessage(ActorCell.scala:577)
        at akka.actor.ActorCell.invoke(ActorCell.scala:547)
        at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:270)
        at akka.dispatch.Mailbox.run(Mailbox.scala:231)
        at akka.dispatch.Mailbox.exec(Mailbox.scala:243)
        at java.util.concurrent.ForkJoinTask.doExec(Unknown Source)
        at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(Unknown Source)
        at java.util.concurrent.ForkJoinPool.runWorker(Unknown Source)
        at java.util.concurrent.ForkJoinWorkerThread.run(Unknown Source)

I am creating the HTTP server in my akka app with

  val handler: HttpRequest => Future[HttpResponse] =
    Route.toFunction(complete(StatusCodes.ImATeapot))

  Http().newServerAt(hostName, port).bind(handler)

(which is according to example Server-Side HTTP/2 (Preview) • Akka HTTP)

If I send the same curl request without option --http2, it works as expected:

curl.exe -v http://localhost:9831
*   Trying 127.0.0.1:9831...
* Connected to localhost (127.0.0.1) port 9831 (#0)
> GET / HTTP/1.1
> Host: localhost:9831
> User-Agent: curl/7.80.0
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 418 I'm a teapot
< Server: NONE-LOCAL-1.0-1.0
< Date: Tue, 21 Dec 2021 10:34:05 GMT
< Content-Type: text/plain; charset=UTF-8
< Content-Length: 49
<
The resulting entity body MAY be short and stout.* Connection #0 to host localhost left intact

Also, if I send the same curl request with switch --http2-prior-knowledge, it seems to work:

curl.exe -v http://localhost:9831 --http2-prior-knowledge
*   Trying 127.0.0.1:9831...
* Connected to localhost (127.0.0.1) port 9831 (#0)
* Using HTTP2, server supports multiplexing
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x132e3b414a0)
> GET / HTTP/2
> Host: localhost:9831
> user-agent: curl/7.80.0
> accept: */*
>
* Connection state changed (MAX_CONCURRENT_STREAMS == 256)!
< HTTP/2 418
< content-type: text/plain; charset=UTF-8
< content-length: 49
< date: Tue, 21 Dec 2021 10:23:21 GMT
< server: NONE-LOCAL-1.0-1.0
<
The resulting entity body MAY be short and stout.* Connection #0 to host localhost left intact

No exception in the log in this case, and the response seems to be HTTP2, according to curl log.

It seems to me as if the connection upgrade process does not seem to work correctly. Or could this be an issue of curl?

I am using

  • AkkaVersion 2.6.12
  • AkkaHttpVersion 10.2.7
  • curl 7.80.0
1 Like

Have this exact same issue as well. There seems to be a fix in the works for 10.2.8, but I don’t know if there is a work-around for 10.2.7.