Lagom1.6.4 Reason for Debug messages : "Resolve (deserialization) of path" and "Delaying cancellation for 1 minute"

nikhilarora · September 30, 2020, 11:00pm

I upgraded my app from Lagom 1.5.4 directly to Lagom 1.6.4. App starts locally in DEV mode and it is able to form a cluster of 3 nodes, but continuously getting lots of DEBUG messages without any traffic on the node. Log messages are mentioned at the end.

Question:
Just want to know the reason for these message and are these serious or can be ignored?
I am afraid that some config is missing or wrong and these messages should not occur continuously.

I am currently testing the app and not even sure if the expected functionality is working properly after upgrade. Will update this post after my test.

Intro about my App

There are 2 Http endpoints and there is 1 one Persistent Entity(UserSubscriptionsEntity) and a ReadProcessor on that entity.
Akka Stream graphs are started once and run for the lifetime of the app and messages are sent from Http endpoint to stream and then to DB and external services.

How I upgraded

I have changed from akka remoting to artery using config
remote {

    artery {
      transport = tcp
      canonical.hostname = ${KUBERNETES_POD_IP}
      canonical.port = ${REMOTING_PORT}
    }
  }

Moved jacksonmodules to akka.serialization.jackson
I was already using ddata for akka.cluster.sharding.store-state-mode

LOGS:

2020-10-01 00:12:08.412 [debug] trp-latest-1-b6m2h [thread-application-akka.actor.default-dispatcher-32] [akka.remote.artery.tcp.ArteryTcpTransport] [akkaAddress=akka://application@100.75.8.156:2552, kamonSpanId=0ef179ccc77958be, sourceThread=application-akka.actor.internal-dispatcher-4, akkaSource=ArteryTcpTransport(akka://application), sourceActorSystem=application, kamonTraceId=20cb3e6ad401167021b3cbfa9156e7fc, kamonSpanName=tell(ActorGraphInterpreter, AsyncInput), akkaTimestamp=22:12:08.412UTC] - Incoming Class Manifest compression advertisement from [akka://application@100.64.11.216:2552#-6504945951225250824], table: [CompressionTable(-2446765118890572070,33,Map(E -> 8, BG -> 9, BD -> 6, N -> 2, J -> 5, A -> 10, at.georgelabs.pushnotificationservice.impl.aggregates.userSubscriptionsEntity.UserSubscriptionsCommand$DeleteSubscription$DeleteSubscriptionReply$SubscriptionNotFound -> 4, M -> 0, BK -> 12, DONE -> 3, BI -> 7, GE -> 1, at.georgelabs.pushnotificationservice.impl.aggregates.userSubscriptionsEntity.UserSubscriptionsCommand$GetActiveSubscriptions$SuccessResult -> 11, d -> 13))]
2020-10-01 00:12:12.476 [debug] trp-latest-1-b6m2h [thread-application-akka.actor.default-dispatcher-28] [akka.http.impl.util.StreamUtils$DelayCancellationStage$$anon$11] [akkaAddress=akka://application@100.75.8.156:2552, kamonSpanId=437e6f2247f14803, sourceThread=application-akka.actor.default-dispatcher-32, akkaSource=StreamUtils$DelayCancellationStage$$anon$11(akka://application), sourceActorSystem=application, kamonTraceId=7a7a4e3181890703021c6538bf111410, kamonSpanName=tell(ActorGraphInterpreter, AsyncInput), akkaTimestamp=22:12:12.476UTC] - Delaying cancellation for 1 minute
2020-10-01 00:12:13.981 [debug] trp-latest-1-b6m2h [thread-application-akka.actor.default-dispatcher-27] [akka.http.impl.util.StreamUtils$DelayCancellationStage$$anon$11] [akkaAddress=akka://application@100.75.8.156:2552, kamonSpanId=f6951ce790d29387, sourceThread=application-akka.actor.default-dispatcher-28, akkaSource=StreamUtils$DelayCancellationStage$$anon$11(akka://application), sourceActorSystem=application, kamonTraceId=013b6fe9acd10a4b566711becab1e7a9, kamonSpanName=tell(ActorGraphInterpreter, AsyncInput), akkaTimestamp=22:12:13.981UTC] - Delaying cancellation for 1 minute
2020-10-01 00:12:22.476 [debug] trp-latest-1-b6m2h [thread-application-akka.actor.default-dispatcher-30] [akka.http.impl.util.StreamUtils$DelayCancellationStage$$anon$11] [akkaAddress=akka://application@100.75.8.156:2552, kamonSpanId=87111eaacb2fbeaa, sourceThread=application-akka.actor.default-dispatcher-10, akkaSource=StreamUtils$DelayCancellationStage$$anon$11(akka://application), sourceActorSystem=application, kamonTraceId=27eb55dade4fe76c4921d535fa48e5ae, kamonSpanName=tell(ActorGraphInterpreter, AsyncInput), akkaTimestamp=22:12:22.476UTC] - Delaying cancellation for 1 minute
2020-10-01 00:12:23.982 [debug] trp-latest-1-b6m2h [thread-application-akka.actor.default-dispatcher-10] [akka.http.impl.util.StreamUtils$DelayCancellationStage$$anon$11] [akkaAddress=akka://application@100.75.8.156:2552, kamonSpanId=38827d81ae4a03e3, sourceThread=application-akka.actor.default-dispatcher-5, akkaSource=StreamUtils$DelayCancellationStage$$anon$11(akka://application), sourceActorSystem=application, kamonTraceId=fb996ac6e71cdf869684e4a752a0cac0, kamonSpanName=tell(ActorGraphInterpreter, AsyncInput), akkaTimestamp=22:12:23.981UTC] - Delaying cancellation for 1 minute
2020-10-01 00:12:32.476 [debug] trp-latest-1-b6m2h [thread-application-akka.actor.default-dispatcher-10] [akka.http.impl.util.StreamUtils$DelayCancellationStage$$anon$11] [akkaAddress=akka://application@100.75.8.156:2552, kamonSpanId=3f5585ff6f4d55da, sourceThread=application-akka.actor.default-dispatcher-31, akkaSource=StreamUtils$DelayCancellationStage$$anon$11(akka://application), sourceActorSystem=application, kamonTraceId=f9dbada4c1b01c7ee1e3d5bf997ab51d, kamonSpanName=tell(ActorGraphInterpreter, AsyncInput), akkaTimestamp=22:12:32.476UTC] - Delaying cancellation for 1 minute
2020-10-01 00:12:33.982 [debug] trp-latest-1-b6m2h [thread-application-akka.actor.default-dispatcher-30] [akka.http.impl.util.StreamUtils$DelayCancellationStage$$anon$11] [akkaAddress=akka://application@100.75.8.156:2552, kamonSpanId=9a729ed826730303, sourceThread=application-akka.actor.default-dispatcher-32, akkaSource=StreamUtils$DelayCancellationStage$$anon$11(akka://application), sourceActorSystem=application, kamonTraceId=0bdbb273f5555cdad9835bdac49c402e, kamonSpanName=tell(ActorGraphInterpreter, AsyncInput), akkaTimestamp=22:12:33.981UTC] - Delaying cancellation for 1 minute
2020-10-01 00:12:42.476 [debug] trp-latest-1-b6m2h [thread-application-akka.actor.default-dispatcher-27] [akka.http.impl.util.StreamUtils$DelayCancellationStage$$anon$11] [akkaAddress=akka://application@100.75.8.156:2552, kamonSpanId=3671cc17c0629fac, sourceThread=application-akka.actor.default-dispatcher-31, akkaSource=StreamUtils$DelayCancellationStage$$anon$11(akka://application), sourceActorSystem=application, kamonTraceId=96adcba7b1bf3308e5dd42993bc9dca0, kamonSpanName=tell(ActorGraphInterpreter, AsyncInput), akkaTimestamp=22:12:42.476UTC] - Delaying cancellation for 1 minute
2020-10-01 00:12:43.981 [debug] trp-latest-1-b6m2h [thread-application-akka.actor.default-dispatcher-28] [akka.http.impl.util.StreamUtils$DelayCancellationStage$$anon$11] [akkaAddress=akka://application@100.75.8.156:2552, kamonSpanId=1abd692b34ce7cef, sourceThread=application-akka.actor.default-dispatcher-31, akkaSource=StreamUtils$DelayCancellationStage$$anon$11(akka://application), sourceActorSystem=application, kamonTraceId=b1e8666a9cc17e7ce6d47aa2acbb8928, kamonSpanName=tell(ActorGraphInterpreter, AsyncInput), akkaTimestamp=22:12:43.981UTC] - Delaying cancellation for 1 minute


2020-10-01 00:04:01.284 [debug] trp-latest-1-7lsm2 [thread-application-akka.actor.default-dispatcher-32] [akka.actor.LocalActorRefProvider.Deserialization] [akkaAddress=akka://application@100.64.11.216:2552, kamonSpanId=5e47bb0622ba3e6c, sourceThread=application-akka.actor.internal-dispatcher-5, akkaSource=akka.actor.LocalActorRefProvider.Deserialization, sourceActorSystem=application, kamonTraceId=a85ea24a66eeaf96ccb8445c4ff01f14, kamonSpanName=tell(ActorGraphInterpreter, AsyncInput), akkaTimestamp=22:04:01.284UTC] - Resolve (deserialization) of path [system/sharding/replicator/$t#-1806007872] doesn't match an active actor. It has probably been stopped, using deadLetters.
2020-10-01 00:04:01.284 [debug] trp-latest-1-7lsm2 [thread-application-akka.actor.default-dispatcher-32] [akka.actor.LocalActorRefProvider.Deserialization] [akkaAddress=akka://application@100.64.11.216:2552, kamonSpanId=5e47bb0622ba3e6c, sourceThread=application-akka.actor.internal-dispatcher-5, akkaSource=akka.actor.LocalActorRefProvider.Deserialization, sourceActorSystem=application, kamonTraceId=a85ea24a66eeaf96ccb8445c4ff01f14, kamonSpanName=tell(ActorGraphInterpreter, AsyncInput), akkaTimestamp=22:04:01.284UTC] - Resolve (deserialization) of path [system/sharding/replicator/$0#888809063] doesn't match an active actor. It has probably been stopped, using deadLetters.
2020-10-01 00:04:01.284 [debug] trp-latest-1-7lsm2 [thread-application-akka.actor.default-dispatcher-32] [akka.actor.LocalActorRefProvider.Deserialization] [akkaAddress=akka://application@100.64.11.216:2552, kamonSpanId=5e47bb0622ba3e6c, sourceThread=application-akka.actor.internal-dispatcher-5, akkaSource=akka.actor.LocalActorRefProvider.Deserialization, sourceActorSystem=application, kamonTraceId=a85ea24a66eeaf96ccb8445c4ff01f14, kamonSpanName=tell(ActorGraphInterpreter, AsyncInput), akkaTimestamp=22:04:01.284UTC] - Resolve (deserialization) of path [system/sharding/replicator/$+#-1452539807] doesn't match an active actor. It has probably been stopped, using deadLetters.
2020-10-01 00:04:01.284 [debug] trp-latest-1-7lsm2 [thread-application-akka.actor.default-dispatcher-32] [akka.actor.LocalActorRefProvider.Deserialization] [akkaAddress=akka://application@100.64.11.216:2552, kamonSpanId=5e47bb0622ba3e6c, sourceThread=application-akka.actor.internal-dispatcher-5, akkaSource=akka.actor.LocalActorRefProvider.Deserialization, sourceActorSystem=application, kamonTraceId=a85ea24a66eeaf96ccb8445c4ff01f14, kamonSpanName=tell(ActorGraphInterpreter, AsyncInput), akkaTimestamp=22:04:01.284UTC] - Resolve (deserialization) of path [system/sharding/replicator/$M#346819729] doesn't match an active actor. It has probably been stopped, using deadLetters.
2020-10-01 00:04:01.284 [debug] trp-latest-1-7lsm2 [thread-application-akka.actor.default-dispatcher-32] [akka.actor.LocalActorRefProvider.Deserialization] [akkaAddress=akka://application@100.64.11.216:2552, kamonSpanId=5e47bb0622ba3e6c, sourceThread=application-akka.actor.internal-dispatcher-5, akkaSource=akka.actor.LocalActorRefProvider.Deserialization, sourceActorSystem=application, kamonTraceId=a85ea24a66eeaf96ccb8445c4ff01f14, kamonSpanName=tell(ActorGraphInterpreter, AsyncInput), akkaTimestamp=22:04:01.284UTC] - Resolve (deserialization) of path [system/sharding/replicator/$9#2049059547] doesn't match an active actor. It has probably been stopped, using deadLetters.
2020-10-01 00:04:01.284 [debug] trp-latest-1-7lsm2 [thread-application-akka.actor.default-dispatcher-32] [akka.actor.LocalActorRefProvider.Deserialization] [akkaAddress=akka://application@100.64.11.216:2552, kamonSpanId=5e47bb0622ba3e6c, sourceThread=application-akka.actor.internal-dispatcher-5, akkaSource=akka.actor.LocalActorRefProvider.Deserialization, sourceActorSystem=application, kamonTraceId=a85ea24a66eeaf96ccb8445c4ff01f14, kamonSpanName=tell(ActorGraphInterpreter, AsyncInput), akkaTimestamp=22:04:01.284UTC] - Resolve (deserialization) of path [system/ddataReplicator/$a#-241382478] doesn't match an active actor. It has probably been stopped, using deadLetters.
2020-10-01 00:04:01.284 [debug] trp-latest-1-7lsm2 [thread-application-akka.actor.default-dispatcher-32] [akka.actor.LocalActorRefProvider.Deserialization] [akkaAddress=akka://application@100.64.11.216:2552, kamonSpanId=5e47bb0622ba3e6c, sourceThread=application-akka.actor.internal-dispatcher-5, akkaSource=akka.actor.LocalActorRefProvider.Deserialization, sourceActorSystem=application, kamonTraceId=a85ea24a66eeaf96ccb8445c4ff01f14, kamonSpanName=tell(ActorGraphInterpreter, AsyncInput), akkaTimestamp=22:04:01.284UTC] - Resolve (deserialization) of path [system/sharding/replicator/$Q#1855415972] doesn't match an active actor. It has probably been stopped, using deadLetters.
2020-10-01 00:04:01.284 [debug] trp-latest-1-7lsm2 [thread-application-akka.actor.default-dispatcher-32] [akka.actor.LocalActorRefProvider.Deserialization] [akkaAddress=akka://application@100.64.11.216:2552, kamonSpanId=5e47bb0622ba3e6c, sourceThread=application-akka.actor.internal-dispatcher-5, akkaSource=akka.actor.LocalActorRefProvider.Deserialization, sourceActorSystem=application, kamonTraceId=a85ea24a66eeaf96ccb8445c4ff01f14, kamonSpanName=tell(ActorGraphInterpreter, AsyncInput), akkaTimestamp=22:04:01.284UTC] - Resolve (deserialization) of path [system/sharding/replicator/$bb#-1644977988] doesn't match an active actor. It has probably been stopped, using deadLetters.
2020-10-01 00:04:01.284 [debug] trp-latest-1-7lsm2 [thread-application-akka.actor.default-dispatcher-32] [akka.actor.LocalActorRefProvider.Deserialization] [akkaAddress=akka://application@100.64.11.216:2552, kamonSpanId=5e47bb0622ba3e6c, sourceThread=application-akka.actor.internal-dispatcher-5, akkaSource=akka.actor.LocalActorRefProvider.Deserialization, sourceActorSystem=application, kamonTraceId=a85ea24a66eeaf96ccb8445c4ff01f14, kamonSpanName=tell(ActorGraphInterpreter, AsyncInput), akkaTimestamp=22:04:01.284UTC] - Resolve (deserialization) of path [system/sharding/replicator/$G#1421946721] doesn't match an active actor. It has probably been stopped, using deadLetters.
2020-10-01 00:04:01.284 [debug] trp-latest-1-7lsm2 [thread-application-akka.actor.default-dispatcher-32] [akka.actor.LocalActorRefProvider.Deserialization] [akkaAddress=akka://application@100.64.11.216:2552, kamonSpanId=5e47bb0622ba3e6c, sourceThread=application-akka.actor.internal-dispatcher-5, akkaSource=akka.actor.LocalActorRefProvider.Deserialization, sourceActorSystem=application, kamonTraceId=a85ea24a66eeaf96ccb8445c4ff01f14, kamonSpanName=tell(ActorGraphInterpreter, AsyncInput), akkaTimestamp=22:04:01.284UTC] - Resolve (deserialization) of path [system/sharding/replicator/$1#234620033] doesn't match an active actor. It has probably been stopped, using deadLetters.
2020-10-01 00:04:01.284 [debug] trp-latest-1-7lsm2 [thread-application-akka.actor.default-dispatcher-32] [akka.actor.LocalActorRefProvider.Deserialization] [akkaAddress=akka://application@100.64.11.216:2552, kamonSpanId=5e47bb0622ba3e6c, sourceThread=application-akka.actor.internal-dispatcher-5, akkaSource=akka.actor.LocalActorRefProvider.Deserialization, sourceActorSystem=application, kamonTraceId=a85ea24a66eeaf96ccb8445c4ff01f14, kamonSpanName=tell(ActorGraphInterpreter, AsyncInput), akkaTimestamp=22:04:01.284UTC] - Resolve (deserialization) of path [system/ddataReplicator/$b#-2121190629] doesn't match an active actor. It has probably been stopped, using deadLetters.
2020-10-01 00:04:01.284 [debug] trp-latest-1-7lsm2 [thread-application-akka.actor.default-dispatcher-32] [akka.actor.LocalActorRefProvider.Deserialization] [akkaAddress=akka://application@100.64.11.216:2552, kamonSpanId=5e47bb0622ba3e6c, sourceThread=application-akka.actor.internal-dispatcher-5, akkaSource=akka.actor.LocalActorRefProvider.Deserialization, sourceActorSystem=application, kamonTraceId=a85ea24a66eeaf96ccb8445c4ff01f14, kamonSpanName=tell(ActorGraphInterpreter, AsyncInput), akkaTimestamp=22:04:01.284UTC] - Resolve (deserialization) of path [system/sharding/replicator/$p#1851847219] doesn't match an active actor. It has probably been stopped, using deadLetters.
2020-10-01 00:04:01.284 [debug] trp-latest-1-7lsm2 [thread-application-akka.actor.default-dispatcher-32] [akka.actor.LocalActorRefProvider.Deserialization] [akkaAddress=akka://application@100.64.11.216:2552, kamonSpanId=5e47bb0622ba3e6c, sourceThread=application-akka.actor.internal-dispatcher-5, akkaSource=akka.actor.LocalActorRefProvider.Deserialization, sourceActorSystem=application, kamonTraceId=a85ea24a66eeaf96ccb8445c4ff01f14, kamonSpanName=tell(ActorGraphInterpreter, AsyncInput), akkaTimestamp=22:04:01.284UTC] - Resolve (deserialization) of path [system/ddataReplicator/$h#-1443581227] doesn't match an active actor. It has probably been stopped, using deadLetters.
2020-10-01 00:04:01.284 [debug] trp-latest-1-7lsm2 [thread-application-akka.actor.default-dispatcher-32] [akka.actor.LocalActorRefProvider.Deserialization] [akkaAddress=akka://application@100.64.11.216:2552, kamonSpanId=5e47bb0622ba3e6c, sourceThread=application-akka.actor.internal-dispatcher-5, akkaSource=akka.actor.LocalActorRefProvider.Deserialization, sourceActorSystem=application, kamonTraceId=a85ea24a66eeaf96ccb8445c4ff01f14, kamonSpanName=tell(ActorGraphInterpreter, AsyncInput), akkaTimestamp=22:04:01.284UTC] - Resolve (deserialization) of path [system/sharding/replicator/$s#-1259736293] doesn't match an active actor. It has probably been stopped, using deadLetters.
2020-10-01 00:04:01.284 [debug] trp-latest-1-7lsm2 [thread-application-akka.actor.default-dispatcher-32] [akka.actor.LocalActorRefProvider.Deserialization] [akkaAddress=akka://application@100.64.11.216:2552, kamonSpanId=5e47bb0622ba3e6c, sourceThread=application-akka.actor.internal-dispatcher-5, akkaSource=akka.actor.LocalActorRefProvider.Deserialization, sourceActorSystem=application, kamonTraceId=a85ea24a66eeaf96ccb8445c4ff01f14, kamonSpanName=tell(ActorGraphInterpreter, AsyncInput), akkaTimestamp=22:04:01.284UTC] - Resolve (deserialization) of path [system/sharding/UserSubscriptionsPushNotificationValidationRepositoryEventProcessorCoordinator/singleton/coordinator/$i#-1790132992] doesn't match an active actor. It has probably been stopped, using deadLetters.
2020-10-01 00:04:01.284 [debug] trp-latest-1-7lsm2 [thread-application-akka.actor.default-dispatcher-32] [akka.actor.LocalActorRefProvider.Deserialization] [akkaAddress=akka://application@100.64.11.216:2552, kamonSpanId=5e47bb0622ba3e6c, sourceThread=application-akka.actor.internal-dispatcher-5, akkaSource=akka.actor.LocalActorRefProvider.Deserialization, sourceActorSystem=application, kamonTraceId=a85ea24a66eeaf96ccb8445c4ff01f14, kamonSpanName=tell(ActorGraphInterpreter, AsyncInput), akkaTimestamp=22:04:01.284UTC] - Resolve (deserialization) of path [system/ddataReplicator/$j#2014798377] doesn't match an active actor. It has probably been stopped, using deadLetters.
2020-10-01 00:04:01.284 [debug] trp-latest-1-7lsm2 [thread-application-akka.actor.default-dispatcher-32] [akka.actor.LocalActorRefProvider.Deserialization] [akkaAddress=akka://application@100.64.11.216:2552, kamonSpanId=5e47bb0622ba3e6c, sourceThread=application-akka.actor.internal-dispatcher-5, akkaSource=akka.actor.LocalActorRefProvider.Deserialization, sourceActorSystem=application, kamonTraceId=a85ea24a66eeaf96ccb8445c4ff01f14, kamonSpanName=tell(ActorGraphInterpreter, AsyncInput), akkaTimestamp=22:04:01.284UTC] - Resolve (deserialization) of path [system/sharding/replicator/$O#-686566758] doesn't match an active actor. It has probably been stopped, using deadLetters.
2020-10-01 00:04:01.284 [debug] trp-latest-1-7lsm2 [thread-application-akka.actor.default-dispatcher-32] [akka.actor.LocalActorRefProvider.Deserialization] [akkaAddress=akka://application@100.64.11.216:2552, kamonSpanId=5e47bb0622ba3e6c, sourceThread=application-akka.actor.internal-dispatcher-5, akkaSource=akka.actor.LocalActorRefProvider.Deserialization, sourceActorSystem=application, kamonTraceId=a85ea24a66eeaf96ccb8445c4ff01f14, kamonSpanName=tell(ActorGraphInterpreter, AsyncInput), akkaTimestamp=22:04:01.284UTC] - Resolve (deserialization) of path [system/sharding/replicator/$D#-581134017] doesn't match an active actor. It has probably been stopped, using deadLetters.
2020-10-01 00:04:01.284 [debug] trp-latest-1-7lsm2 [thread-application-akka.actor.default-dispatcher-32] [akka.actor.LocalActorRefProvider.Deserialization] [akkaAddress=akka://application@100.64.11.216:2552, kamonSpanId=5e47bb0622ba3e6c, sourceThread=application-akka.actor.internal-dispatcher-5, akkaSource=akka.actor.LocalActorRefProvider.Deserialization, sourceActorSystem=application, kamonTraceId=a85ea24a66eeaf96ccb8445c4ff01f14, kamonSpanName=tell(ActorGraphInterpreter, AsyncInput), akkaTimestamp=22:04:01.284UTC] - Resolve (deserialization) of path [system/sharding/UserSubscriptionsEntityCoordinator/singleton/coordinator/$e#-1935545077] doesn't match an active actor. It has probably been stopped, using deadLetters.
2020-10-01 00:04:01.284 [debug] trp-latest-1-7lsm2 [thread-application-akka.actor.default-dispatcher-32] [akka.actor.LocalActorRefProvider.Deserialization] [akkaAddress=akka://application@100.64.11.216:2552, kamonSpanId=5e47bb0622ba3e6c, sourceThread=application-akka.actor.internal-dispatcher-5, akkaSource=akka.actor.LocalActorRefProvider.Deserialization, sourceActorSystem=application, kamonTraceId=a85ea24a66eeaf96ccb8445c4ff01f14, kamonSpanName=tell(ActorGraphInterpreter, AsyncInput), akkaTimestamp=22:04:01.284UTC] - Resolve (deserialization) of path [system/ddataReplicator/$c#-1004260068] doesn't match an active actor. It has probably been stopped, using deadLetters.
2020-10-01 00:04:01.284 [debug] trp-latest-1-7lsm2 [thread-application-akka.actor.default-dispatcher-32] [akka.actor.LocalActorRefProvider.Deserialization] [akkaAddress=akka://application@100.64.11.216:2552, kamonSpanId=5e47bb0622ba3e6c, sourceThread=application-akka.actor.internal-dispatcher-5, akkaSource=akka.actor.LocalActorRefProvider.Deserialization, sourceActorSystem=application, kamonTraceId=a85ea24a66eeaf96ccb8445c4ff01f14, kamonSpanName=tell(ActorGraphInterpreter, AsyncInput), akkaTimestamp=22:04:01.284UTC] - Resolve (deserialization) of path [system/sharding/UserSubscriptionsEntity/38/7902699BE42C8A8E46FBBB4501726517E86B22C56A189F7625A6DA49081B2451#-1492869252] doesn't match an active actor. It has probably been stopped, using deadLetters.
2020-10-01 00:04:01.284 [debug] trp-latest-1-7lsm2 [thread-application-akka.actor.default-dispatcher-32] [akka.actor.LocalActorRefProvider.Deserialization] [akkaAddress=akka://application@100.64.11.216:2552, kamonSpanId=5e47bb0622ba3e6c, sourceThread=application-akka.actor.internal-dispatcher-5, akkaSource=akka.actor.LocalActorRefProvider.Deserialization, sourceActorSystem=application, kamonTraceId=a85ea24a66eeaf96ccb8445c4ff01f14, kamonSpanName=tell(ActorGraphInterpreter, AsyncInput), akkaTimestamp=22:04:01.284UTC] - Resolve (deserialization) of path [system/sharding/replicator/$ab#326517392] doesn't match an active actor. It has probably been stopped, using deadLetters.
2020-10-01 00:04:01.284 [debug] trp-latest-1-7lsm2 [thread-application-akka.actor.default-dispatcher-32] [akka.actor.LocalActorRefProvider.Deserialization] [akkaAddress=akka://application@100.64.11.216:2552, kamonSpanId=5e47bb0622ba3e6c, sourceThread=application-akka.actor.internal-dispatcher-5, akkaSource=akka.actor.LocalActorRefProvider.Deserialization, sourceActorSystem=application, kamonTraceId=a85ea24a66eeaf96ccb8445c4ff01f14, kamonSpanName=tell(ActorGraphInterpreter, AsyncInput), akkaTimestamp=22:04:01.284UTC] - Resolve (deserialization) of path [system/sharding/UserSubscriptionsEntityCoordinator/singleton/coordinator/$j#90917852] doesn't match an active actor. It has probably been stopped, using deadLetters.
2020-10-01 00:04:01.284 [debug] trp-latest-1-7lsm2 [thread-application-akka.actor.default-dispatcher-32] [akka.actor.LocalActorRefProvider.Deserialization] [akkaAddress=akka://application@100.64.11.216:2552, kamonSpanId=5e47bb0622ba3e6c, sourceThread=application-akka.actor.internal-dispatcher-5, akkaSource=akka.actor.LocalActorRefProvider.Deserialization, sourceActorSystem=application, kamonTraceId=a85ea24a66eeaf96ccb8445c4ff01f14, kamonSpanName=tell(ActorGraphInterpreter, AsyncInput), akkaTimestamp=22:04:01.284UTC] - Resolve (deserialization) of path [system/sharding/replicator/$V#149182968] doesn't match an active actor. It has probably been stopped, using deadLetters.
2020-10-01 00:04:01.284 [debug] trp-latest-1-7lsm2 [thread-application-akka.actor.default-dispatcher-32] [akka.actor.LocalActorRefProvider.Deserialization] [akkaAddress=akka://application@100.64.11.216:2552, kamonSpanId=5e47bb0622ba3e6c, sourceThread=application-akka.actor.internal-dispatcher-5, akkaSource=akka.actor.LocalActorRefProvider.Deserialization, sourceActorSystem=application, kamonTraceId=a85ea24a66eeaf96ccb8445c4ff01f14, kamonSpanName=tell(ActorGraphInterpreter, AsyncInput), akkaTimestamp=22:04:01.284UTC] - Resolve (deserialization) of path [system/sharding/replicator/$X#582039585] doesn't match an active actor. It has probably been stopped, using deadLetters.
2020-10-01 00:04:01.284 [debug] trp-latest-1-7lsm2 [thread-application-akka.actor.default-dispatcher-32] [akka.actor.LocalActorRefProvider.Deserialization] [akkaAddress=akka://application@100.64.11.216:2552, kamonSpanId=5e47bb0622ba3e6c, sourceThread=application-akka.actor.internal-dispatcher-5, akkaSource=akka.actor.LocalActorRefProvider.Deserialization, sourceActorSystem=application, kamonTraceId=a85ea24a66eeaf96ccb8445c4ff01f14, kamonSpanName=tell(ActorGraphInterpreter, AsyncInput), akkaTimestamp=22:04:01.284UTC] - Resolve (deserialization) of path [system/sharding/UserSubscriptionsEntityCoordinator/singleton/coordinator/$c#-1122324654] doesn't match an active actor. It has probably been stopped, using deadLetters.
2020-10-01 00:04:01.284 [debug] trp-latest-1-7lsm2 [thread-application-akka.actor.default-dispatcher-32] [akka.actor.LocalActorRefProvider.Deserialization] [akkaAddress=akka://application@100.64.11.216:2552, kamonSpanId=5e47bb0622ba3e6c, sourceThread=application-akka.actor.internal-dispatcher-5, akkaSource=akka.actor.LocalActorRefProvider.Deserialization, sourceActorSystem=application, kamonTraceId=a85ea24a66eeaf96ccb8445c4ff01f14, kamonSpanName=tell(ActorGraphInterpreter, AsyncInput), akkaTimestamp=22:04:01.284UTC] - Resolve (deserialization) of path [system/sharding/replicator/$v#-1873554656] doesn't match an active actor. It has probably been stopped, using deadLetters.
2020-10-01 00:04:01.284 [debug] trp-latest-1-7lsm2 [thread-application-akka.actor.default-dispatcher-32] [akka.actor.LocalActorRefProvider.Deserialization] [akkaAddress=akka://application@100.64.11.216:2552, kamonSpanId=5e47bb0622ba3e6c, sourceThread=application-akka.actor.internal-dispatcher-5, akkaSource=akka.actor.LocalActorRefProvider.Deserialization, sourceActorSystem=application, kamonTraceId=a85ea24a66eeaf96ccb8445c4ff01f14, kamonSpanName=tell(ActorGraphInterpreter, AsyncInput), akkaTimestamp=22:04:01.284UTC] - Resolve (deserialization) of path [system/sharding/UserSubscriptionsPushNotificationValidationRepositoryEventProcessorCoordinator/singleton/coordinator/$e#727703242] doesn't match an active actor. It has probably been stopped, using deadLetters

ignasi35 · October 1, 2020, 8:32am

Hi @nikhilarora

I’ve edited your post and tagged under the akka-cluster category aswell.

These are debug level logs. I don’t think you need to worry about them.

nikhilarora · October 1, 2020, 10:44am

Thanks Ignasi

I see this code from which the message is logged.
Questions is: Who is sending messages to the actors in local JVM that by the time message reaches, the actor is already dead ?

 /**
   * INTERNAL API
   */
  private[akka] def resolveActorRef(ref: InternalActorRef, pathElements: Iterable[String]): InternalActorRef =
    if (pathElements.isEmpty) {
      logDeser.debug("Resolve (deserialization) of empty path doesn't match an active actor, using deadLetters.")
      deadLetters
    } else
      ref.getChild(pathElements.iterator) match {
        case Nobody =>
          if (log.isDebugEnabled)
            logDeser.debug(
              "Resolve (deserialization) of path [{}] doesn't match an active actor. " +
              "It has probably been stopped, using deadLetters.",
              pathElements.mkString("/"))
          new EmptyLocalActorRef(system.provider, ref.path / pathElements, eventStream)
        case x => x
      }

nikhilarora · October 3, 2020, 8:55pm

I found that the messages Delaying cancellation for 1 minute is related to linger-timeout = 1 min in akka-http-core

Topic		Replies	Views
No acknowledgement received when trying to register to coordinator, messages buffered. Lagom Persistence API akka-cluster , configuration	0	1412	April 23, 2020
Scheduled sending of heartbeat was delayed Lagom akka , akka-cluster	2	1128	September 22, 2019
Lagom service warning "Sending an 2xx 'early' response before end of request was received" Lagom akka-http	4	719	November 16, 2018
Integrating Play with Lagom Play Framework java , akka-cluster , lagom	3	1922	May 16, 2018
Lagom 1.4.11 - Association failed Caused by: [The remote system explicitly disassociated (reason unknown) Lagom	12	3357	January 21, 2020

Lagom1.6.4 Reason for Debug messages : "Resolve (deserialization) of path" and "Delaying cancellation for 1 minute"

Related Topics