Decoding a chunked gzip-compressed request

Akka Akka HTTP
1

Hi all,

I am trying to set up a connection between an Akka HTTP Client and an Akka HTTP Server using a chunked+gzip transfer encoding. On the client side, I am using .via(Compression.gzip) and manually setting the Transfer-Encoding: gzip header as the docs say there is no high-level support for this.

On the server side, I am using decodeRequest { extractDataBytes { ... } }. My assumption was that this would take care of both decompression and dechunking, but when I’m running the sample below (Akka HTTP 10.1.3) the data bytes are still gzipped in my inner route. Am I missing something obvious?

Thanks!

object ChunkedGzipTest {
  def main(args: Array[String]) {

    implicit val system = ActorSystem("my-system")
    implicit val materializer = ActorMaterializer()
    implicit val executionContext = system.dispatcher

    val route =
      post {
        decodeRequest {
          extractDataBytes { dataSource =>
            onSuccess(dataSource.runFold(ByteString.empty)(_.concat(_))) { bs =>
              println(bs.decodeString("UTF-8"))
              complete(HttpEntity("OK"))
            }
          }
        }
      }

    val bindingFuture = Http().bindAndHandle(route, "localhost", 9999)

    val testEntity: MessageEntity = HttpEntity.Chunked
            .fromData(ContentTypes.`text/plain(UTF-8)`, Source(immutable.Seq(ByteString("This is my teststring")))
                    .via(Compression.gzip))
    val transferEncHeaders = immutable.Seq(`Transfer-Encoding`(TransferEncodings.gzip))
    val response = Http.get(system)
            .singleRequest(HttpRequest(HttpMethods.POST, Uri("http://localhost:9999"), transferEncHeaders, testEntity))
    Await.result(response, Duration.Inf)
    bindingFuture.flatMap(_.unbind()).onComplete(_ => system.terminate())
  }
}
2

I found the problem that seems to cause my issue here. The server-side decodeRequest handles only a Content-Encoding header, but not a Transfer-Encoding header in the request.

Is there a way to correctly handle both cases transparently on the server side? Or do I need to manually inspect the Transfer-Encoding header?

3

Hi @schoeneu,

Transfer-Encoding is handled specially in akka-http (through the choice of HttpEntity subclasses). Setting the header manually usually has no effect. On the other hand, akka-http does not support gzip Transfer-Encoding itself.

However…

You should probably use gzip Content-Encoding which is what the decodeRequest will work with. Also Compression.gzip is the akka-streams level gzip support. We also have akka-http level gzip support which will directly set the right headers.

So, on the client side, you can use

akka.http.scaladsl.coding.Gzip.encodeMessage(uncompressedRequest)

to create an encoded request.

Here’s a complete working example:

import akka.actor.ActorSystem
import akka.http.scaladsl.Http
import akka.http.scaladsl.model.ContentTypes
import akka.http.scaladsl.model.HttpEntity
import akka.http.scaladsl.model.HttpMethods
import akka.http.scaladsl.model.HttpRequest
import akka.http.scaladsl.server.Directives._
import akka.stream.ActorMaterializer
import akka.stream.scaladsl.Source
import akka.util.ByteString
import com.typesafe.config.ConfigFactory

import scala.concurrent.Await
import scala.concurrent.duration._

object ChunkedGzipTest {
  def main(args: Array[String]) {

    val config =
      ConfigFactory.parseString(
        """
          |akka.loglevel = debug
          |akka.http.server.log-unencrypted-network-bytes = 1000
          |akka.http.client.log-unencrypted-network-bytes = 1000
        """.stripMargin
      ).withFallback(ConfigFactory.defaultApplication())

    implicit val system = ActorSystem("my-system", config)
    implicit val materializer = ActorMaterializer()
    implicit val executionContext = system.dispatcher

    val route =
      post {
        decodeRequest {
          entity(as[String]) { text ⇒
            println(s"Server got: $text")
            complete(s"Got: $text")
          }
        }
      }

    val bindingFuture = Http().bindAndHandle(route, "localhost", 9999)

    val testDataSource = Source("This is " :: "my" :: " teststring" :: Nil map (ByteString(_, "utf8")))
    val uncompressedRequest =
      HttpRequest(
        HttpMethods.POST,
        uri = "http://localhost:9999",
        entity = HttpEntity.Chunked.fromData(ContentTypes.`text/plain(UTF-8)`, testDataSource)
      )
    val compressedRequest = Gzip.encodeMessage(uncompressedRequest)

    val responseFut = Http(system).singleRequest(compressedRequest)
    val response = Await.result(responseFut, 10.seconds)
    println("Got response")
    println(response)

    bindingFuture.flatMap(_.unbind()).onComplete(_ ⇒ system.terminate())
  }
}

Hope that helps!

1 Like
4

Thanks! Since I have control over both ends of this channel, I was able to migrate everything to Content-Encoding like you suggested.

1 Like