Access Cassandra on Kubernetes

Hi,

I’m trying to run online-auction-scala example on kubernates. Kubernates is installed as standalone on Linux box (so, it is not minikube).

kubectl get service:

NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
bidding ClusterIP 10.110.88.131 10000/TCP,10001/TCP,10002/TCP 27h
cassandra ClusterIP 10.97.53.213 9042/TCP 19h
item ClusterIP 10.109.50.248 10000/TCP,10001/TCP,10002/TCP 27h
kafka ClusterIP 10.109.125.211 9092/TCP 28h
kafka-headless ClusterIP None 9092/TCP 28h
kafka-zookeeper ClusterIP 10.108.234.104 2181/TCP 28h
kafka-zookeeper-headless ClusterIP None 2181/TCP,3888/TCP,2888/TCP 28h
kubernetes ClusterIP 10.96.0.1 443/TCP 28h

kubectl exec -it busybox – nslookup cassandra.default.svc.cluster.local

Server: 10.96.0.10

Address: 10.96.0.10:53

Name: cassandra.default.svc.cluster.local

Address: 10.97.53.213

*** Can’t find cassandra.default.svc.cluster.local: No answer

A command to deploy bidding-imp:

rp generate-kubernetes-resources localhost:5000/bsi/biddingimpl:1.0.0-SNAPSHOT --generate-all --registry-disable-https --registry-disable-tls-validation
–pod-controller-replicas 2
–env JAVA_OPTS="-Dplay.crypto.secret=youmustchangeme4"
–env CASSANDRA_SERVICE_NAME=“cassandra.default.svc.cluster.local” | kubectl apply -f -

Then I find these errors in logs:

com.example.auction.bidding.impl.AuctionScheduler [] - Error running finish bidding query

com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: /10.97.53.213:0 (com.datastax.driver.core.exceptions.TransportException: [/10.97.53.213:0] Cannot connect))

Did I configure CASSANDRA_SERVICE_NAME well?
Does lagom trying to access Cassandra on port 0 ?

Zlaja

It’s expecting you to configure with a full SRV domain name, which would include the correct port in the response.

This would look something like _cql._tcp.cassandra.default.svc.cluster.local, but the port name (_cql in this example) might be different depending on how you’ve configured your Cassandra Kubernetes Service.

I spend a few hours on this (deploy Lagom online auction with Cassandra on Kubernetes) and here is my rp command, hope it helps:

#Please note, cassandra is the service name in the Kubernetes namespace called taylor

export service_cassandra=_cql._tcp.cassandra.taylor.svc.cluster.local

#Deploy bidding-impl

rp generate-kubernetes-resources some.domain.of.image.host/biddingimpl:1.0.0
–generate-pod-controllers --generate-services
–env JAVA_OPTS="-Dplay.http.secret.key=$secret_bidding -Dplay.filters.hosts.allowed.0=$allowed_host"
–pod-controller-replicas 2
–registry-use-local
–transform-pod-controllers ‘.spec.template.spec.imagePullSecrets += [{name:“scalefunction-registry-secrets”}]’
–external-service “cas_native=$service_cassandra”
–external-service “kafka_native=$service_kafka” | kubectl apply -f -

Hi Zuoky

Can you Please also paste the $service_kafka here as well

Regards
Qasim Raza

Hi,

You can check example definition in Lagom Auction - Deploy projects

DNS SRV for kafka, cassandra or any other “external” service that is configured using DNS SRV will depend on the “external” service configuration.
Example configuration for cassandra running outside of kubernetes cluster using headless service:

apiVersion: v1
kind: Service
metadata:
  name: cassandra
  namespace: msrp
spec:
  ports:
  - name: "cql"
    protocol: "TCP"
    port: 9042
    targetPort: 9042
    nodePort: 0

---
apiVersion: v1
kind: Endpoints
metadata:
 name: cassandra
 namespace: msrp
subsets:
 - addresses:
     - ip: 10.0.1.6
     - ip: 10.0.2.93
     - ip: 10.0.3.99
   ports:
     - name: "cql"
       port: 9042

So for this configuration DNS SRV will be:
_cql._tcp.cassandra.msrp.svc.cluster.local

Check DNS for Services and Pods - DNS SRV for details

Hope this helps.

BR,
Alan