Resolve attempt failed! Cause: spray.json.DeserializationException


(Joo) #1

I’m keep getting this exception when trying to launch one of my services in the Google Kunbernetes engine.

Not sure what is missing here. I do have package dependencies configured for all the JSON codecs and etc.

2018-05-09T03:59:04.163Z [e[33mwarne[0m] akka.management.cluster.bootstrap.dns.HeadlessServiceDnsBootstrap [sourceThread=application-akka.actor.default-dispatcher-17, akkaTimestamp=03:59:04.163UTC, akkaSource=akka.tcp://application@10.16.1.30:10001/system/headlessServiceDnsBootstrap, sourceActorSystem=application] - Resolve attempt failed! Cause: spray.json.DeserializationException: Object is missing required member 'items'


(Tim Moore) #2

This looks like an issue with the Akka Management cluster bootstrap process, rather than your service code itself or something in Lagom. I’m going to move the thread over to an Akka category to get the right eyes on it :eyes:


(Joo) #3

Thanks a lot Tim.

I tried this with the Lagom’s online-auction-scala just now. Deployed to the same Google KBE cluster and getting the same issue.

Just created the issue here:


(Joo) #4

To provide more details, I see lots of warnings like this when I SBT build this project. Same for online-auction-scala. I am wondering if this has something to do with this version conflicts.

[warn] Found version conflict(s) in library dependencies; some are suspected to be binary incompatible:
[warn] 	* io.netty:netty-handler:4.1.22.Final is selected over {4.0.56.Final, 4.1.13.Final, 4.0.44.Final}
[warn] 	    +- com.lightbend.lagom:lagom-client_2.12:1.4.4        (depends on 4.1.22.Final)
[warn] 	    +- com.datastax.cassandra:cassandra-driver-core:3.2.0 (depends on 4.0.44.Final)
[warn] 	    +- com.lightbend.lagom:lagom-persistence-cassandra-core_2.12:1.4.4 (depends on 4.0.44.Final)
[warn] 	    +- com.typesafe.netty:netty-reactive-streams:2.0.0    (depends on 4.1.13.Final)
[warn] 	    +- com.datastax.cassandra:cassandra-driver-core:3.5.0 (depends on 4.0.56.Final)
[warn] 	* com.github.jnr:jnr-constants:0.9.9 is selected over 0.9.0
[warn] 	    +- org.lmdbjava:lmdbjava:0.6.0                        (depends on 0.9.9)
[warn] 	    +- com.github.jnr:jnr-posix:3.0.44                    (depends on 0.9.9)
[warn] 	    +- com.github.jnr:jnr-posix:3.0.27                    (depends on 0.9.0)
[warn] 	* io.netty:netty:3.10.6.Final is selected over 3.10.5.Final
[warn] 	    +- com.typesafe.akka:akka-remote_2.12:2.5.11 ()       (depends on 3.10.6.Final)
[warn] 	    +- org.apache.zookeeper:zookeeper:3.4.10              (depends on 3.10.5.Final)
[warn] 	* com.google.guava:guava:22.0 is selected over {16.0.1, 19.0, 16.0}
[warn] 	    +- com.lightbend.lagom:lagom-scaladsl-play-json_2.12:1.4.4 (depends on 22.0)
[warn] 	    +- com.lightbend.lagom:lagom-api_2.12:1.4.4           (depends on 22.0)
[warn] 	    +- com.typesafe.play:play_2.12:2.6.12                 (depends on 22.0)
[warn] 	    +- com.lightbend.lagom:lagom-scaladsl-cluster_2.12:1.4.4 (depends on 22.0)
[warn] 	    +- com.fasterxml.jackson.datatype:jackson-datatype-guava:2.8.11 (depends on 16.0)
[warn] 	    +- com.datastax.cassandra:cassandra-driver-core:3.2.0 (depends on 19.0)
[warn] 	    +- com.datastax.cassandra:cassandra-driver-extras:3.5.0 (depends on 19.0)
[warn] 	    +- com.datastax.cassandra:cassandra-driver-core:3.5.0 (depends on 19.0)
[warn] 	    +- org.apache.curator:curator-test:2.10.0             (depends on 16.0.1)
[warn] 	    +- org.apache.curator:curator-client:2.10.0           (depends on 16.0.1)
[warn] 	* com.typesafe.akka:akka-stream_2.12:2.5.11 is selected over {2.4.20, 2.5.9, 2.5.7}
[warn] 	    +- com.typesafe.akka:akka-remote_2.12:2.5.11 ()       (depends on 2.5.11)
[warn] 	    +- com.lightbend.lagom:lagom-api_2.12:1.4.4           (depends on 2.5.11)
[warn] 	    +- com.typesafe.akka:akka-persistence-query_2.12:2.5.11 () (depends on 2.5.11)
[warn] 	    +- com.typesafe.play:play-streams_2.12:2.6.12         (depends on 2.5.11)
[warn] 	    +- com.lightbend.akka.management:akka-management_2.12:0.9.0 (depends on 2.5.7)
[warn] 	    +- com.lightbend.akka.discovery:akka-discovery-marathon-api_2.12:0.9.0 (depends on 2.5.7)
[warn] 	    +- com.lightbend.akka.discovery:akka-discovery-kubernetes-api_2.12:0.9.0 (depends on 2.5.7)
[warn] 	    +- com.typesafe.akka:akka-stream-kafka_2.12:0.18      (depends on 2.5.7)
[warn] 	    +- com.typesafe.play:play-ws-standalone_2.12:1.1.6    (depends on 2.5.9)
[warn] 	    +- com.typesafe.akka:akka-http-core_2.12:10.0.11 ()   (depends on 2.4.20)
[warn] 	* com.typesafe.akka:akka-actor_2.12:2.5.11 is selected over {2.5.7, 2.4.20}
[warn] 	    +- com.typesafe.akka:akka-remote_2.12:2.5.11 ()       (depends on 2.5.11)
[warn] 	    +- com.lightbend.lagom:lagom-api_2.12:1.4.4           (depends on 2.5.11)
[warn] 	    +- com.typesafe.akka:akka-stream_2.12:2.5.11 ()       (depends on 2.5.11)
[warn] 	    +- com.typesafe.akka:akka-testkit_2.12:2.5.11 ()      (depends on 2.5.11)
[warn] 	    +- com.lightbend.lagom:lagom-scaladsl-play-json_2.12:1.4.4 (depends on 2.5.11)
[warn] 	    +- com.typesafe.akka:akka-persistence_2.12:2.5.11 ()  (depends on 2.5.11)
[warn] 	    +- com.typesafe.play:play_2.12:2.6.12                 (depends on 2.5.11)
[warn] 	    +- com.typesafe.akka:akka-slf4j_2.12:2.5.11 ()        (depends on 2.5.11)
[warn] 	    +- com.typesafe.akka:akka-parsing_2.12:10.0.11 ()     (depends on 2.4.20)
[warn] 	    +- com.lightbend.rp:reactive-lib-async-dns_2.12:0.7.0 (depends on 2.5.7)
[warn] 	    +- com.lightbend.akka.discovery:akka-discovery_2.12:0.9.0 (depends on 2.5.7)
[warn] 	    +- com.lightbend.akka.management:akka-management_2.12:0.9.0 (depends on 2.5.7)
[warn] 	    +- com.lightbend.akka.discovery:akka-discovery-marathon-api_2.12:0.9.0 (depends on 2.5.7)
[warn] 	    +- com.lightbend.akka.discovery:akka-discovery-kubernetes-api_2.12:0.9.0 (depends on 2.5.7)
[

(Joo) #5

I just managed to get it working, but it is only a temporal solution I guess.

You just have to disable the akkaClusterBootstrap in the build.sbt, and make the replications factor to 1.

So in build.sbt:

enableAkkaClusterBootstrap := false

and then rp goes like:

rp generate-kubernetes-resources gcr.io/chainp/instrument-impl:1.0.0 \
  --generate-pod-controllers --generate-services \
  --env JAVA_OPTS="-Dplay.http.secret.key=$secret_key" \
  --pod-controller-replicas 1 \
  --external-service "cas_native=$service_cassandra" \
  --external-service "kafka_native=$service_kafka" | kubectl apply -f -

(Tim Moore) #6

Hmm… I wouldn’t expect that solution to work for services that actually do require a cluster (I don’t know if that applies to yours).

We have had another user report the same symptom, also on Google Cloud, so it isn’t just you. On the other hand, other people have used Google Cloud successfully so it’s not completely incompatible either! :smile:

I’ll investigate…


(Tim Moore) #7

I think I found the problem. Please see my comment on the GitHub issue you raised: https://github.com/lagom/online-auction-scala/issues/78#issuecomment-387933707