I would like to share an observation we have had with the use of Akka HTTP client in a context where an F5 reverse proxy distributes traffic to 2 different Play Servers (let’s call them A and B), internally listening resp. on the ports 9000 and 9001, but both sharing the same public hostname. The redirect happens based on the local path of the HTTP request on that public host. The F5 is a full proxy and sets up separate connections for the client and the servers.
Public requests use HTTPS, the TLS is terminated on the F5 and internally the Play servers are using the HTTP protocol, not HTTPS.
The use case is a sequence where first a GET request is submitted to server A (internally routed to an IP address and port 9000), followed by another request - a POST request - to the same public host, which should then redirect to server B (internally routed to an IP address and port 9001).
The first request checks if the user is eligible for POST’ing to server B and if so, it tries to POST a message to server B, provided server A confirms the user can be posted to.
The observation is:
- The first request runs successfully to completion (GET request to server A)
- The second POST however fails, with a message (POST request to server B)
Outgoing request stream error akka.stream.StreamTcpException: The connection closed with error: An existing connection was forcibly closed by the remote host
In the server B logs, we find the following message
Illegal request, responding with status ‘400 Bad Request’: Unsupported HTTP method: The HTTP method started with 0x16 rather than any known HTTP method. Perhaps this was an HTTPS request sent to an HTTP endpoint?
It is worth noting that this problem only presents itself with the Akka HTTP Client, perhaps due to the way the connection pool works ?
Our application uses Http().singleRequest. Another setup with a dedicated host connection pool did not resolve it, nor a setup with a Source queue, but the result is the same.
Before thinking it might be something with the F5, it is worth noting that we can successfully execute this sequence with other Scala HTTP clients (we did a test with STTP and a few other backends, and with these it is working fine. Although our application uses the Akka HTTP client, not SoftwareMill’s, when we switched to STTP with Akka Streams as backend, we noticed that, we see the same error pop up.
Could someone explain why we are making this observation. Is there some special care needed when using Akka HTTP client ? Am I missing something ?
For us this seems to indicate a specific issue that exists only with Akka HTTP client.