How to adjust the max upload chunk size in Play?

configuration

(Mark Hammons) #1

I’ve implemented a streaming uploader using a custom body parser for my play website that can take n-byte uploads. It’s meant to be used to upload files ranging from between 2MB and 70GB. Right now, it uses a chunk size of around 1MB, but i’d like to allow 4-16MB chunks. When I try to use chunks bigger than 1MB, I get this error:

akka.http.scaladsl.model.EntityStreamException: HTTP chunk size exceeds the configured limit of 1048576 bytes. How do I fix this?

Here’s my BodyParser implementation:

  def streamingUploader(experimentName: ExperimentName,
                        aid: String,
                        group: String,
                        fileName: String) = BodyParser { implicit req =>
    val path = aid match {
      case "0" => (IRODSPath(experimentName) / group / fileName).throwFault
      case n   => ???
    }

    val expInfo = pm.tryGetExperimentStubByName(experimentName)

    val auth = PTryT(expInfo).flatMapF { exp =>
      authModule.forceUser(OwnedBy(exp.owner.userID, exp.owningTeam), Write)
    }

    val future = auth
      .flatMapF { _ =>
        logger.info(s"creating directories for $path")
        path.getParent.mkDirs()
      }
      .flatMapF { _ =>
        path.createWriter(4 * 1024 * 1024, overwrite = true)
      }
      .map { writer =>
        logger.info(s"creating sink for $path")
        Sink
          .fold[PTry[(Writer, Long)], ByteString](PSuccess(writer, 0l)) {
            case (mW, bs: ByteString) =>
              mW.flatMap {
                case (w, bytes) => w.write(bs).map(_ -> (bytes + bs.size))
              }
          }
          .mapMaterializedValue { res =>
            PTryT(res).map(_._2).value.map { r =>
              writer.close()
              r
            }
          }
      }
      .map { iSink =>
        logger.info(s"creating accumulator for $path")
        Accumulator(iSink)
          .map {
            case PFailure(f) =>
              Left(InternalServerError(f.faultTrace.mkString("\n")))
            case PSuccess(bs) =>
              logger.info(s"done")
              Right(bs)
          }
      }
      .leftMap { f =>
        logger.error(f.faultTrace.mkString("\n"))
        //todo: create a fault handler spec, and get the appropriate response with it
        Accumulator.done(Left(InternalServerError(f.faultTrace.mkString("\n"))))
      }
      .fold(identity, identity)

    Await.result(future, 3.minutes)
  }

(Aditya Athalye) #2

You might want to check the max-chunk-size configuration for the Akka HTTP server here. The default is 1M.

https://doc.akka.io/docs/akka-http/current/configuration.html


(Mark Hammons) #3

I’ve added the following to my application.conf:

akka.http {
	server{
		parsing{
			max-chunk-size = 16MB
		}
	}

	parsing {
		max-chunk-size = 16MB
	}

	client {
		parsing {
			max-chunk-size = 16MB
		}
	}
}

still no good.


(Aditya Athalye) #4

This issue seems to be related to what you are seeing, unless you have already checked it out.


(Mark Hammons) #5

I think it is. I am using sttp for the upload. Problem is that the sttp client isn’t throwing the error. The play server is. I’ve set max-chunk-size for 3 contexts to try to override play (as you can see above), and I still get the error. Not really sure what I’m doing wrong.


(Aditya Athalye) #6

Well then the best option to go for is to debug the server code and see where the exception emanates from. Generally works :slight_smile:

Optionally you can get hold of the entire config object on the server to make sure your settings took effect.


(Mark Hammons) #7

I’ll have to give that a shot tomorrow.


(Mark Hammons) #8

One last thing, do you know why I’d be getting this error?

2018-09-29 20:21:50,681 [warn] a.a.ActorSystemImpl - Sending an 2xx 'early' response before end of request was received... Note that the connection will be closed after this response. Also, many clients will not read early responses! Consider only issuing this response after the request data has been completely read!


(Aditya Athalye) #9

I debugged the AkkaHttpServer.scala and confirmed that it does not pick up your setting.
So if you go back to the play documentation, you will find an important footnote.

Which means, your changes in application.conf are not going to be used as the server would have started by then.
As recommended, if you are running your server in dev mode, add any properties in build.sbt.

Try this in your build.sbt

PlayKeys.devSettings += "akka.http.parsing.max-chunk-size" -> "16m"

This setting changed the httpconfig loaded by the AkkaHttpServer. This might work for you.


(Mark Hammons) #10

This does indeed solve the problem for me! Thanks for the help :slight_smile:

The only issue I have left is the “early response” warning that I’m getting when the upload finishes. Dunno why play says that I’m responding early though…