Error: You are creating too many HashedWheelTimer instances

Play Framework
1

Hi,

I’m getting this error in the log:

You are creating too many HashedWheelTimer instances. HashedWheelTimer is a shared resource that must be reused across the JVM, so that only a few instances are created.

The error appears in the log after about a week of using the application in production.
The processor usage goes up about 1 percent each day. After a week it reaches 10% and won’t go down. I haven’t seen if it goes up further, since we are deploying a new version every week.

I can’t find what I’m doing wrong.

We are using play 2.8.7

Controller example:

public class MyController extends Controller {

final ActorSystem actorSystem;

final DatabaseExecutionContext dbEc;

@Inject
public MyController(ActorSystem actorSystem, DatabaseExecutionContext dbEc) {
    this.actorSystem = actorSystem;
    this.dbEc = dbEc;
}

public CompletionStage<Result> analysisList(){
    return CompletableFuture.supplyAsync(()->
            ok(Json.toJson(Analysis.find.all())),
            HttpExecution.fromThread((Executor) dbEc));
}

public CompletionStage<Result> performLongRunningTask(Http.Request request){
    final Map<String, String[]> params = request.body().asFormUrlEncoded();
    
    if (params.containsKey("analysisId")) {
        return CompletableFuture.supplyAsync(() -> {
            
            Analysis item = Analysis.find.byId(params.get("analysisId")[0]);
            
            if (item != null && item.isReady()) {

                actorSystem.scheduler().scheduleOnce(Duration.ZERO, () -> longRunningTask(item), dbEc);

                return ok("Task in process");
            }
            
            return ok("Item not ready");

        }, HttpExecution.fromThread((Executor) dbEc));
    }else 
        return CompletableFuture.completedFuture(badRequest("Missing parameter"));
}

private void longRunningTask(Analysis item){
    
    //Do something that takes a long time to complete.
    //High database usage.

}

public class DatabaseExecutionContext extends CustomExecutionContext {

@Inject
public DatabaseExecutionContext(ActorSystem actorSystem) {

    super(actorSystem, "database.dispatcher");
}

}

Application.conf

database.dispatcher{
type = Dispatcher
executor = “thread-pool-executor”
throughput = 1
thread-pool-executor{
fixed-pool-size = 20
}
}

2

If I remember correctly, HashedWheelTimer is an internal class to Netty. Review what netty usages are there on you application: it may be the Play’s HTTP backend (but the default HTTP backend in Play 2.8.x is Akka HTTP), it may be a database driver, or maybe you use a third party library which brings Netty as a transitive dependency.

It looks like you are creating too many instances, instead of reusing a single one or releasing the instances once used.

Cheers,

3

You are absolutely right:

“It looks like you are creating too many instances, instead of reusing a single one or releasing the instances once used.”

I was not looking in the right place. After searching through the code I found the piece of code that was not releasing resources.

Example code:

private final WSClient client;
private final ActorSystem system;

public MyClass() {

String name = "wsclient";
system = ActorSystem.create(name);
Materializer materializer = Materializer.matFromSystem(system);
AsyncHttpClientConfig asyncHttpClientConfig =
        new DefaultAsyncHttpClientConfig.Builder()
                .setMaxRequestRetry(0)
                .setShutdownQuietPeriod(0)
                .setShutdownTimeout(0)
                .build();
AsyncHttpClient asyncHttpClient = new DefaultAsyncHttpClient(asyncHttpClientConfig);
client = new AhcWSClient(asyncHttpClient, materializer);

}
/***
Some other methods
***/

Then I didn’t release the resources correctly. I added:

client.close();
system.terminate();

The problem is gone.

Thanks so much.

4

Why are you creating a new instance of the ActorSystem and the AhcWSClient for each instance of MyClass? You should be able to inject those into the class and share them across your entire application. Both the ActorSystem and AsyncHttpClient are expensive to create so you want to avoid creating many instances of them.

1 Like
5

Hi Greg,

Thanks for your comments. MyModule is extending Abstract Module, so I couldn’t use @Inject. After some research I found that I could use @Provides on my method. The MyModule class now looks like this:

public class MyModule extends AbstractModule {

@Override
protected void configure() {

//Some code

}

@Provides
@Singleton
protected SomeOtherClass myMethod(WSClient wsClient){

return new SomeOtherClass(“String parameter”,new MyClass(wsClient));

}

}

public MyClass implements AnotherClass{

private final WSClient wsClient;

@Inject
public MyClass(WSClient wsClient) {
this.wsClient = wsClient;
}

@Override
public void method_A(ParameterClass parameter){

//Do something with wsClient

}

}