Unfathomable NotEnoughDataException

Hi everyone, I’ve been stuck on this issue for a while now, but unsure if it’s a bug, and even if it’s really play causing it.

My production setup is a Play 2.8.5 installation behind a Nginx proxy on an up to date Ubuntu 18.04 server; it only happens there, and not in local development (even when also proxied via a local nginx)

For a small but important part of our application, the client side code sends a POST request to the server containing a form encoded as multipart-formdata, consisting of a single multiple selection field witch possibly many (>1000) selected values. Upon receiving such request, the play server has the relevant action process these values, constructing either a string or bytearray (depending on the url), and returning these with the corresponding mime type etc.

Now for the weird problem: if I select few values, I can repeatedly get a successful response to my requests; likewise, if I request a large number of values followed by a smaller number, or a small number followed by a larger number of values, there seems to be no problem; however, as soon as I send multiple requests containing a large number of values (even if I wait for the first one to finish before sending the next), I get the correct response to the first, but the server crashes on the second with a NotEnoughDataException. This obscure exception, about which I can hardly find any info, occurs in the processing of the MultipartFormData (stack trace can be found at pastebin.com/E2JPQuPB).

Because of its request-size-dependent behavior, I thought it would be a buffer issue, but I increased play.http.parser.maxMemoryBuffer and play.http.parser.maxDiskBuffer to multiple times the request size, without avail. Moreover, the first request always returns ok, no matter it’s size, and I thought requests are supposed to be independent on one another (and thus the buffer used by the first request would not be used for the second request).

Another strange aspect of the error is that the NotEnoughDataException only becomes fatal because it leads to a NoClassDefFoundError, meaning it could not initialize the NotEnoughDataException class. This is obvious, since it is, in fact, an object. But when looking to the code, I cannot find any place where it refers to it as a class.

To summarize: I can not fathom what is happening here, and since it happens before any of our own code, I don’t know how to diagnose it further.

Any ideas would be much appreciated!

Kind regards,

Wouter

I had a look at the stack trace. I am pretty sure you are not running latest Play 2.8.5

        at play.core.parsers.Multipart$BodyPartParser$$anon$1.tryParseInitialBoundary(Multipart.scala:348)
        at play.core.parsers.Multipart$BodyPartParser$$anon$1.$anonfun$state$1(Multipart.scala:310)
        at play.core.parsers.Multipart$BodyPartParser$$anon$1.onPush(Multipart.scala:315)

The above lines only fit to Play 2.8.0 or 2.8.1. E.g. in Play 2.8.1 Multipart.scala:348 would be this line, however with Play 2.8.2 the lines referred in the stack trace don’t fit anymore. Where/how did you configure your application that makes you think your are running Play 2.8.5?

Also

Could not initialize class play.core.parsers.Multipart$NotEnoughDataException$

makes me believe you maybe have multiple Play jars on your classpath that interfere with each other?
How do you deploy your application?
It doesn’t make sense that NotEnoughDataException can’t be initialized, it’s defined in the same file like the code that calls it. Something must be wrong.

Oh, my bad, the trace indeed was from when I first encountered the problem, which was before we updated to 2.8.5. However, the issue remains the same (I just did not notice the change in line numbers). Here’s a more recent trace, for what it’s worth: pastebin.com/83JMPQ0G.

As you say, @mkurz , it does indeed not make any sense that it cannot initialize the exception. That’s what been puzzling me, and why I thought about raising it as a bug on github. I just wanted to see if the someone here saw something I didn’t.

Where/how did you configure your application that makes you think your are running Play 2.8.5? [The error] makes me believe you maybe have multiple Play jars on your classpath that interfere with each other? How do you deploy your application?

To answer to your questions: I use sbt with Play’s sbt-plugin; I build the project and deploy it as a jar on our remote server. I ran sbt evicted to check for conflicting versions, but it automatically prefers 2.8.5 over 2.8.0 (which is a dependency of the latest version of enumerated). I also cleaned my Ivy cache before doing a clean build to make sure sbt really pulled 2.8.5.

Do you see anything wrong? Something I forgot? Suggestions as to what I might try different?

Thanks in advance!

How do you build the jar?

That shouldn’t be necessary. However make sure you run clean within sbt (or just sbt clean) before building the jar (or just remove any target folders within your project, also the one inside the project folder)

How do you run the jar on the server?

I build the jar using sbt dist. I know sbt clean should suffice to remove any unwanted previous versions — it was one of the first things I did — but apparently is still not suffices.

On the server, the jar is simply called by a systemd service executing java -jar with some memory parameters, a port, and reference to a config file … nothing of which should influence the version of Play.

Update: we now circumvented the issue by using form-urlencoded instead of multipart/formdata. Of course, that doesn’t answer what went wrong in the first place.

Actually here is an explanation what is happening: StackOverflowError or NoClassDefFoundError is occurred when received form-data with many fields from multipart/form-data · Issue #10590 · playframework/playframework · GitHub