Not able to start the Shard coordinator

Hi Team
We are facing the following problem when we start the cluster with 2 nodes:

Seed Node Configuration:

remote {
    log-remote-lifecycle-events = off
    netty.tcp {
      hostname = 127.0.0.1
      port = 2555
    }
  }
cluster {
    jmx.multi-mbeans-in-same-jvm = on
    sharding.state-store-mode = persistence
    sharding.remember-entities = on
    seed-nodes = [
      "akka.tcp://users-actor-system@127.0.0.1:2555"]

    roles = ["AuthEntityServer","SessionEntityServer"]
    metrics.enabled = off
  }

Joining Node configuration

remote {
    log-remote-lifecycle-events = off
    netty.tcp {
      hostname = 127.0.0.1
      port = 2551
    }
  }
 cluster {
    jmx.multi-mbeans-in-same-jvm = on
    sharding.state-store-mode = persistence
    sharding.remember-entities = on
    seed-nodes = [
      "akka.tcp://users-actor-system@127.0.0.1:2555"]

    roles = ["UserEntityServer","GroupEntityServer","UserUniquenessCheckServer"]
    metrics.enabled = off
  }

akka-persistence-cassandra version : 0.80-RC1
akka version : 2.5.6

Starting order:
Seed node is started first and “Joining Node” is started after that.

Problem:
When the “Joining Node” Starts 3 shards using ClusterSharding(as).start(), the seed node is reporting the below information (3*3 times). Also noticed that the coordinator for the 3 shards are not created(not seeing a row getting inserted in to cassandra table).
Is there anything I am missing in the configuration?

Errors:

85201 [users-actor-system-akka.actor.default-dispatcher-18] INFO  akka.actor.EmptyLocalActorRef - Message [akka.cluster.sharding.ShardCoordinator$Internal$Register] from Actor[akka.tcp://users-actor-system@127.0.0.1:2551/system/sharding/GroupEntity#1559239295] to Actor[akka://users-actor-system/system/sharding/GroupEntityCoordinator/singleton/coordinator] was not delivered. [2] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. 

85202 [users-actor-system-akka.actor.default-dispatcher-18] INFO  akka.actor.EmptyLocalActorRef - Message [akka.cluster.sharding.ShardCoordinator$Internal$Register] from Actor[akka.tcp://users-actor-system@127.0.0.1:2551/system/sharding/UserEntity#-885384634] to Actor[akka://users-actor-system/system/sharding/UserEntityCoordinator/singleton/coordinator] was not delivered. [3] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. 

85204 [users-actor-system-akka.actor.default-dispatcher-18] INFO  akka.actor.EmptyLocalActorRef - Message [akka.cluster.sharding.ShardCoordinator$Internal$Register] from Actor[akka.tcp://users-actor-system@127.0.0.1:2551/system/sharding/ApplicationUniqueFieldCheckerSchemaEntity#1728683026] to Actor[akka://users-actor-system/system/sharding/ApplicationUniqueFieldCheckerSchemaEntityCoordinator/singleton/coordinator] was not delivered. [4] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. 

87186 [users-actor-system-akka.actor.default-dispatcher-17] INFO  akka.actor.EmptyLocalActorRef - Message [akka.cluster.sharding.ShardCoordinator$Internal$Register] from Actor[akka.tcp://users-actor-system@127.0.0.1:2551/system/sharding/UserEntity#-885384634] to Actor[akka://users-actor-system/system/sharding/UserEntityCoordinator/singleton/coordinator] was not delivered. [5] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. 

87188 [users-actor-system-akka.actor.default-dispatcher-17] INFO  akka.actor.EmptyLocalActorRef - Message [akka.cluster.sharding.ShardCoordinator$Internal$Register] from Actor[akka.tcp://users-actor-system@127.0.0.1:2551/system/sharding/GroupEntity#1559239295] to Actor[akka://users-actor-system/system/sharding/GroupEntityCoordinator/singleton/coordinator] was not delivered. [6] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. 

87203 [users-actor-system-akka.actor.default-dispatcher-17] INFO  akka.actor.EmptyLocalActorRef - Message [akka.cluster.sharding.ShardCoordinator$Internal$Register] from Actor[akka.tcp://users-actor-system@127.0.0.1:2551/system/sharding/ApplicationUniqueFieldCheckerSchemaEntity#1728683026] to Actor[akka://users-actor-system/system/sharding/ApplicationUniqueFieldCheckerSchemaEntityCoordinator/singleton/coordinator] was not delivered. [7] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. 

89185 [users-actor-system-akka.actor.default-dispatcher-17] INFO  akka.actor.EmptyLocalActorRef - Message [akka.cluster.sharding.ShardCoordinator$Internal$Register] from Actor[akka.tcp://users-actor-system@127.0.0.1:2551/system/sharding/UserEntity#-885384634] to Actor[akka://users-actor-system/system/sharding/UserEntityCoordinator/singleton/coordinator] was not delivered. [8] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. 

89187 [users-actor-system-akka.actor.default-dispatcher-17] INFO  akka.actor.EmptyLocalActorRef - Message [akka.cluster.sharding.ShardCoordinator$Internal$Register] from Actor[akka.tcp://users-actor-system@127.0.0.1:2551/system/sharding/GroupEntity#1559239295] to Actor[akka://users-actor-system/system/sharding/GroupEntityCoordinator/singleton/coordinator] was not delivered. [9] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. 

89202 [users-actor-system-akka.actor.default-dispatcher-18] INFO  akka.actor.EmptyLocalActorRef - Message [akka.cluster.sharding.ShardCoordinator$Internal$Register] from Actor[akka.tcp://users-actor-system@127.0.0.1:2551/system/sharding/ApplicationUniqueFieldCheckerSchemaEntity#1728683026] to Actor[akka://users-actor-system/system/sharding/ApplicationUniqueFieldCheckerSchemaEntityCoordinator/singleton/coordinator] was not delivered. [10] dead letters encountered, no more dead letters will be logged. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. 

Thank you
Anand Krishnan

We also tried the following version of Akka and Cassandra journal plugin and got the same results:
akka-persistence-cassandra version : 0.84
akka version : 2.5.12

Are you starting cluster sharding on the seed node as well?

I am not performing above action on seed node for GroupEntity, UserEntity and ApplicationUniqueFieldCheckerSchemaEntity shards; we dont want to host all shards in all nodes.

Full code example available here - https://github.com/abdheshkumar/akkapersistence-example/tree/master

If you don’t want to host shards on all nodes but are using roles to select a subset you need to tell sharding about that role through the ClusterShardingSettings

Thanks Johan this helped

val actorRef = ClusterSharding(as).start(
      typeName = shardActorName,
      entityProps = props(as, producer),
      settings = ClusterShardingSettings(system = as).withRole(role),  //Adding a role here
      extractEntityId = entityIdExtractor,
      extractShardId = entityShardResolver)

Thank you
Anand