AskTimeOut exception, local/fast Kafka/Cassandra


(James Harlow) #1

I am getting

akka.pattern.AskTimeoutException: Ask timed out on [Actor[akka://x-impl-application/system/sharding/XEntity#-1074623916]] after [30000 ms]. Sender[null] sent message of type "com.lightbend.lagom.scaladsl.persistence.CommandEnvelope"

when running my service tests, and when curling against sbt runAll, with a call stack that has none of my code in it. [1] (I was getting this with 5s timeouts, and upped them). When I make requests that throw exceptions in the event handler (and I guess then doesn’t persist?), the service responds very quickly.

I’m working against a local Kafka, which is almost instant when I insert pairs with kafka-console-producer, and the command handler simply copies the command data to an event and then persists it - I don’t see how it could take 30s. Simplified (but not very simplified!) code below. This is the first command that’s being sent to an entity. Is there any way to measure how long the entity takes to initialize?

Any suggestions on how to debug or even where to look next?

[1]

16:48:23.081 [debug] com.elevenfs.foundry.payments.impl.XServiceImpl [] - asked to persist an A event
16:48:23.082 [debug] com.elevenfs.foundry.payments.impl.XServiceImpl [] - about to persist an A event
16:48:23.085 [debug] com.elevenfs.foundry.payments.impl.XServiceImpl [] - persisted an A event
16:48:53.048 [error] myService [] - Exception in PathCallIdImpl(/api/a/:id)
akka.pattern.AskTimeoutException: Ask timed out on [Actor[akka://x-impl-application/system/sharding/XEntity#1400171378]] after [30000 ms]. Sender[null] sent message of type "com.lightbend.lagom.scaladsl.persistence.CommandEnvelope".
	at akka.pattern.PromiseActorRef$.$anonfun$defaultOnTimeout$1(AskSupport.scala:596)
	at akka.pattern.PromiseActorRef$.$anonfun$apply$1(AskSupport.scala:606)
	at akka.actor.Scheduler$$anon$4.run(Scheduler.scala:205)
	at scala.concurrent.Future$InternalCallbackExecutor$.unbatchedExecute(Future.scala:866)
	at scala.concurrent.BatchingExecutor.execute(BatchingExecutor.scala:109)
	at scala.concurrent.BatchingExecutor.execute$(BatchingExecutor.scala:103)
	at scala.concurrent.Future$InternalCallbackExecutor$.execute(Future.scala:864)
	at akka.actor.LightArrayRevolverScheduler$TaskHolder.executeTask(LightArrayRevolverScheduler.scala:328)
	at akka.actor.LightArrayRevolverScheduler$$anon$4.executeBucket$1(LightArrayRevolverScheduler.scala:279)
	at akka.actor.LightArrayRevolverScheduler$$anon$4.nextTick(LightArrayRevolverScheduler.scala:283)
	at akka.actor.LightArrayRevolverScheduler$$anon$4.run(LightArrayRevolverScheduler.scala:235)
	at java.lang.Thread.run(Thread.java:748)

[2]

class XEntity extends PersistentEntity {
  override def initialState: State = None

  override def behavior: Behavior = {
    case None =>
      Actions()
        .onCommand[A_cmd, Done] {
          case (A_cmd(a, b, c, d), ctx, _) =>
            logger.debug("asked to persist an A event")
            ctx.thenPersist(
              A_evt(a, b, c, d)
            ) { _ =>
              logger.debug("persisted an A event")
              ctx.done
            }
        }
        .onEvent {
          case (A_evt(a, b, c, d), _) =>
            logger.debug("about to persist an A event")
            Some(S(a, b, c, d, None, "first_state"))
        }

(Alan Klikic) #2

@hythloday i think you miss ctx.reply before ctx.done indicating you are not sending response on your command and getting ask timeout


(James Harlow) #3

Of course I am. That fixed it. Thank you so much!


(Alan Klikic) #4

Np.
I had this so many times ;)
Also one other note if you use or will use persistAll…if you persist an empty list of events reply callback after persist will not be invoked and response will not be sent. So in case of using persistAll always check if your list is empty


(James Harlow) #5

That’s a great tip!


(Patrik Nordwall) #6

empty list of events reply callback after persist will not be invoked

That is an oversight in my opinion and should be fixed. I created issue #1600. Contributions are welcome.