What is the right way to configure cassandra in prod


(Gervais Siewe) #1

I’m running a lagom service on Kubernetes. In the Kubernetes Cluster i have 3 Pods of Cassandra up and running.
Here is how i configured Cassandra in my lagom service:

cassandra-journal.keyspace=authenticationservice
cassandra-snapshot-store.keyspace=authenticationservice
lagom.persistence.read-side.cassandra.keyspace=authenticationservice
lagom.persistence.read-side.cassandra.replication-factor=3
lagom.persistence.read-side.cassandra.write-consistency=“ONE”
lagom.persistence.read-side.cassandra.read-consistency=“ONE”
lagom.persistence.read-side.cassandra.log-queries=true
lagom.persistence.read-side.cassandra.max-result-size=1000

Somehow this configuration doesn’t seem to be correct. Today Morning after one of the cassandra pod restarted, my lagom service what not able to connect to cassandra anymore, i started getting following error:

Caused by: com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency ONE (1 required but only 0 alive)
at com.datastax.driver.core.exceptions.UnavailableException.copy(UnavailableException.java:128)
at com.datastax.driver.core.Responses$Error.asException(Responses.java:126)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:507)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1075)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:998)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286)

Can someone please help me to find the right way to configure Cassandra in my service, what are the configuration i have to properly configure?

Thank you


(Tim Moore) #2

This error seems to indicate that the service can’t find the Cassandra cluster.

How is the service locating Cassandra? Have you configured more than one contact point? If not, was the one that died the same one that was configured?