Play framework streaming responses causes too many open files errors

streams

(Tarik) #21

Hi.If I make load test like this, connections stay open.As you can see I do nothing with the response as a client.

wsClient.url(“http://localhost:9000/downloadTestFile2”).withMethod(“GET”).stream()

But if I make load test like this system is good.No open TCP connections.(I see consume succeed log at this code)

wsClient.url(“http://localhost:9000/downloadTestFile2”).withMethod(“GET”).stream().map{res =>
if(res._1.status == 200){
if(res._1.headers.nonEmpty) {
logger.info(res._1.headers.keys.mkString(" ####### “))
val sink = Sink.ignore
res._2.runWith(sink).onComplete{
case success =>logger.info(”------------consume succeed------------")
case fail =>logger.error("-----------consume failed----------")
}
} else{
logger.info("bytestr : "+res._2.take(150))
}
}else
logger.info("status err : "+res._1.status)
}

If client consume the response in any way, system is ok.But If client just make requests and do nothing, framework doesn’t handle open connections.Don’t think client as only scala code.It may be C#,python,java etc.This is a serious problem for systems.I expect you to simulate this case and give some work around soulution advise.


(Tarik) #22

Is there any progress ?


(Greg Methvin) #23

What you described seems normal to me. If you call stream() without consuming or cancelling the source, then the stream will be backpressured until you read it. That’s one of the caveats of using streaming APIs. So you just shouldn’t do that. If you use stream(), always do something with the stream. As you suggested, this can not happen only with the Play WS client, but also with any other client that keeps a connection open without reading anything.

On the server side, it seems like Play is just doing what you’re asking it to do. It has no way of automatically distinguishing a normal slow connection from one that was unintentionally or maliciously left open. If you are concerned about this, Play does provide an idle timeout setting (play.server.http.idleTimeout) you can adjust to close connections that have been left open without any activity. This is something you have to tune based on your server resources and app. You can also do rate limiting using a reverse proxy like nginx or a third-party library like play-guard, or use a service like CloudFlare.


(Tarik) #24

Thanks for your answer.I want to say something.

This is not a case I created.Someone who knows this bug(I still think this is a bug) can attack to the system.This is a web application and we can not say to end user “please don’t use our system in this way”.I still expect you to simulate this case and say certain solution about this case.At least you can change your official documentation.Because this is a big problem and I think I gave you enough arguments to explain it.I also gave you sample code examples which are simple and clear to understand.This case worths to be examined deeply.

I will try your solution advises.

It seems to solve the problem.I will try it and write the result.


(Greg Methvin) #25

I’m just saying if you’re a client, don’t do that. You’re right that on the server side you shouldn’t just trust your users.

The attack you mentioned is a relatively well-studied problem. A common variation is the slowloris attack, which works by slowly sending small amounts of data over a lot of connections. The good news is that Play is pretty good at handling large numbers of concurrent connections (well over 100k with the right configuration), so it can handle this type of attack much better than servlet-based apps that require a thread per request.

What would be a “solution” to you? Have you tried other web frameworks that handle this case in a better way?


(Tarik) #26

play.server.http.idleTimeout configuration did not work.

My solution expectation is simple.There are hanged TCP connections on my system and how I can clear them.If you read all of my posts from beginning to end you will understand what I mean(especially seventh post).Is there any other configuration you can suggest about idleTimeout or any other.My play version is 2.5.10.I did not try any other framework.


(Marcos Pereira) #27

Hi @tariksahin4391,

There are two idleTimeout settings available for Play 2.5.x:

  1. play.server.http.idleTimeout (or play.server.https.idleTimeout): Play will close connections where both reader and writer have been idle for the specified time. And will do that only in that case.
  2. play.ws.ahc.idleConnectionInPoolTimeout: this is used when your Play application is requesting data from other service using WSClient. It will also timeout only when there is not activity on both ends.

Based on the code you provided at your first post, looks like you want to timeout Play WS, not the Play server. Is that right?

As I said above, Play will understand the connection is “hanged” if there is no activity on both client and server, which is the correct way to consider a connection as idle.

Could you please test the play.ws.ahc.idleConnectionInPoolTimeout on the application that has the downloadFile action and see how it goes?

Best.


(Greg Methvin) #28

@marcospereira My understanding is he wants to prevent clients from being able to DoS his server by opening lots of connections.

@tariksahin4391 In case you don’t know, you need special configuration to make server configuration work with the run command: https://www.playframework.com/documentation/2.5.x/ConfigFile#Using-with-the-run-command


(Tarik) #29

I tried but it didn’t work.I can see it works for the other TCP connections but doesn’t work for TCP connections I mentioned above.They are hanged on the system and never disappear.When I repeat the requests, after some time application doesn’t make TCP connections because there is no awailable port.Maybe these connections are not idle for the system or there is something else.Can you give any other advise(even at operating system level).

I tried in run command and also in build.sbt as PlayKeys.devSettings.It didn’t work.

Don’t think as only attack.Users can download files in many ways.They can use a program code like scala,java etc.The problem is not lots of connections.The problem is hanged connections on the system that never disappear.Also if you have any advise to prevent download requests except from web browser I can listen.

I gave you some sample codes.All you need is to create two project,run the sample codes,follow the steps I mentioned above and you can see the results yourself.If this is a bug (I think it is), I think this is a serious problem.If there is something wrong with my code so you may change your official documentation.Because I developed my code depends on it.


(Tarik) #30

I will try this scenario on play 2.6.x.Maybe 2.6 versions can solve the problem.But upgrade 2.5.10 to 2.6 is a big step for production.I still need a workaround solution.I will write the results.