Cluster Sharding messages to /system/sharding/replicator not delivered

Akka Akka Cluster
,
1

Akka version 2.6.10
Components:
akka-management_2.12
akka-management-cluster-bootstrap_2.12
akka-discovery_2.12
akka-discovery-aws-api_2.12
akka-cluster_2.12
akka-persistence-typed_2.12
akka-cluster-sharding-typed_2.12
…

I am trying to use EventSourcedBehavior with Akka Cluster Sharding running on AWS using EC2 tag-based discovery with my service running in docker (one container per node).

Relevant config:

  discovery {
    aws-api-ec2-tag-based {
      tag-key = "platform:role"
      filters = "tag:MyFilterTag=myfiltertag"
    }
  }

  management {
    cluster.bootstrap.contact-point-discovery {
      service-name = "msg-orchestrator"
      discovery-method = "aws-api-ec2-tag-based"
    }

    http.hostname = ${PRIVATE_IP}
    http.port = 8558
    http.bind-hostname = 0.0.0.0
    http.bind-port = 8558
  }

  remote.artery {
    canonical.port = 25520
    canonical.hostname = ${PRIVATE_IP}
    bind.hostname = 0.0.0.0 # internal (bind) hostname
    bind.port = 25520        # internal (bind) port
  }

  cluster {
    # seed-nodes = ["akka://MsgOrchestrator@127.0.0.1:25520"] - using bootstrap
    sharding {
      number-of-shards = 1000
      downing-provider-class = "akka.cluster.sbr.SplitBrainResolverProvider"
      state-store-mode = ddata
      min-nr-of-members = 2 
    }
  }

Startup code:

        final ClusterSharding sharding = ClusterSharding.get(actorSystem);
        AkkaManagement.get(actorSystem).start();
        ClusterBootstrap.get(actorSystem).start();

Running two nodes (separate boxes), I have gotten to the point where a Leader is elected and both the leader and the other box join the cluster. Both boxes are logging Received gossip status messages.

Where I am stuck now is that, when the non-leader executes a Command (instantiating an EntityRef), the ShardRegion actor starts logging

Trying to register to coordinator at [ActorSelection[Anchor(akka://MsgOrchestrator@<leader_ip>:25520/), Path(/system/sharding/MoMeetingCoordinator/singleton/coordinator)]], but no acknowledgement.

and the leader starts logging

Message [akka.cluster.ddata.Replicator$Internal$Status] from Actor[akka://MsgOrchestrator@<non_leader_ip>:25520/system/sharding/replicator#-502570199] to Actor[akka://MsgOrchestrator/system/sharding/replicator] was not delivered

If I send a Command to the leader that pertains to the EntityRef created on the non-leader, the leader doesn’t know about it.

Any help much appreciated.

UPDATE: Once the “oldest” member of the cluster receives any Command, the ShardCoordinator seems to be initialized, and from thereon, it works as expected. Short of sending a dummy command on cluster startup, is there a correct way to ensure this is initialized? I guess I would have expected the BootstrapCoordinator to take care of this.

2

You need to “start” the specific shard type by calling `ClusterSharding#init(…) during startup on each node. See third sample snippet down in the docs.

3

Thanks - finally figured out that the .init method was global and not to start up a single Behavior instance.