How is the code that performs this calculation invoked? Is it part of a persistent entity command processing, or executed outside an entity?
Lagom doesn’t currently provide a way to run entities on a different dispatcher, but in general we would recommend avoiding very expensive computations within the command handler if possible. Does the computation both depend on the current state of the entity and emit new events to change the state? If you don’t need the current state, then you can move the computation to before the command is sent. If you do need the current state but don’t need to change it, then you can use a read-only command to get the state and the perform the simulation outside the entity. Another possibility might be to move the simulation to a read-side processor. If none of these options are suitable, another option is to customize the cluster dispatcher so that it isn’t blocked by computations running on the default dispatcher.
Being able to customize the dispatcher that Lagom persistent entities use would be a nice enhancement, but it looks like it would require a change to the framework.
And also, my understanding of Akka gossiping protocol is that Akka always spares a fixed number of threads for heartbeat exchange to prevent this heartbeat interval growing issue. So I am not quite sure why this error is happening first of all.
This will become true in Akka 2.6, but isn’t in current stable versions. Right now, the default dispatcher is used for the heartbeats unless you configure it otherwise as described in the link above.
Even with a dedicated cluster dispatcher, blocking the default dispatcher is undesirable. It won’t cause the same cluster heartbeat stability issues, but could result in unpredictable performance in other areas of your application.