Serving files (returning input stream) causes Timeout waiting for connection from pool

Need help with debugging an issue : Caused by: org.apache.http.conn.ConnectionPoolTimeoutException: Timeout waiting for connection from pool
Application based on PlayFramework 2.6. We have Controller class which lets user download files from block storage.

GET API function to downlaod the file

public CompletionStage<Result> fileDownload(String devId, String folder, DocumentId docId, String latest) {
        return downloaFileFromStorage(devId, folder, docId, null, latest)
                .thenApply(downloadableDocumentResponse -> getDownloadResult(docId, null, downloadableDocumentResponse));
    }
private Result getDownloadResult(DocumentId docId, String fileName, Response<DownloadableDoct> userDownloadResponse) {
            ...
            String destinationFileName = String.format("%s.%s", docId.getName(), docId.getExtension()));
            DownloadableDoc userDownload = userDownloadResponse.getData();
            final Result result = commonService.prepareResult(userDownload.getInputStream(), userDownload.getContentLength(), userDownload.getContentType(), docId, destinationFileName);
            ...
        return result;
    }    

public Result prepareResult(InputStream inputStream, long contentLength, String contentType, DocumentId docId, String fileName) {
         return resultWithFileName(((contentLength > 0) ? ok(inputStream, contentLength) : ok(inputStream)), fileName)
                .as(contentType)
                .withHeader(DocumentConstants.DOCUMENT_ID, docId.getDocumentId().toLowerCase());
    }

 public Result resultWithFileName(Result result, String filenameToSaveAs) {
        String encodedFileName = HttpHeaderParameterEncoding.encode("filename", filenameToSaveAs);
        final String contentHeader = String.format(DocumentConstants.ATTACHMENT_CLAUSE, encodedFileName);
        return result.withHeader(DocumentConstants.CONTENT_DISPOSITION, contentHeader);
    }

Http Connectin pool settings

PoolingHttpClientConnectionManager connectionManager = new PoolingHttpClientConnectionManager();
connectionManager.setMaxTotal(2); 
connectionManager.setDefaultMaxPerRoute(1); 

RequestConfig requestConfig = RequestConfig.custom()
        //.setConnectionRequestTimeout(1_000) // when commented this line application gets hung, otherwise it throws ConnectionPoolTimeoutException due to pool exhaustion
        .setSocketTimeout(30_000)
        .setConnectTimeout(30_000)
        .build();
CloseableHttpClient httpClient = HttpClientBuilder.create()
        .setDefaultRequestConfig(requestConfig)
        .setConnectionManager(connectionManager)
        .build();

Also wrapping up stream to log when stream is opened and closed

static class InputStreamTrace extends FilterInputStream {
        private final String message;
        private final Logger logger;

        private InputStreamTrace(InputStream inputStream, String message, Logger logger) {
            super(inputStream);
            this.message = message;
            this.logger = logger;
            logger.info(Thread.currentThread().getName()+" - open_stream : " +message);
        }

        @Override
        public void close() throws IOException {
            logger.info(Thread.currentThread().getName()+" - close_stream : " + message);
            super.close();
        }
    }

When I run this application with above connection pool setting and inputstream wrapper class, After serving few download request application just gets hung.
It opens up a steam and that stream gets is returned to by Controller class to play framework. PlayFramework never closes the inputstream so connection never returns to pool.
Other threads keep waiting on connection pool for a connection and application appears to be hung.

Logs:

  • xxxxx - application-akka.stream.default-blocking-io-dispatcher-126 - close_stream : xxx:\xxxx\xxxx\15e1e20d291baa6
  • xxxxx - application-akka.actor.default-dispatcher-34 - open_stream : xxx:\xxxx\xxxx\15e1e20d291baa6
    • ContentLength: 77063 bytes
    • xxxx - application-akka.actor.default-dispatcher-27 - open_stream : xxx:\xxxx\xxxx\15e1e20d2992c02

At last line in the log it opens an stream and application just gets hung. File size is 77KB so it shouldn’t take long to read the file.
The application which called the file download API times out after 60 secs because it didn’t receive any response.

Enabling debug logging in PlayFramework (Akka) is not helping.
Please help if anybody knows how PlayFramework (Akka) handles blocking io and how to trace why it is not closing or reading input stream.

Few months back faced same issue and content length of 0 was causing same issue. There are files of zero byte size and when an input stream with zero content length is given to PlayFramework it never closes the stream. We handled this case in the code as shown above method “prepareResult”. Issue is once again occurring and now content length is not zero. What happens to an InputStream when it is return by controller to PlayFramework?

Thanks in advance