Akka cluster's first node is special?

There has one akka cluster including three nodes. e.g

“CONFIG_akka_cluster_seedNodes_0=akka.tcp://scheduler-actor-system@IP_A:7000”,
“CONFIG_akka_cluster_seedNodes_1=akka.tcp://scheduler-actor-system@IP_B:7001”,
“CONFIG_akka_cluster_seedNodes_2=akka.tcp://scheduler-actor-system@IP_C:7002”,

Then, on seedNodes_2, send actor message to seedNodes_0, seedNodes_1, seedNodes_2, e.g.

val path = s"akka.tcp://scheduler-actor-system@{endpoints.asAkkaEndpoint}/user/{name}"
val actorSelection = context.actorSelection(path)
actorSelection.ask(…)

In theory, the three nodes are symmetric, but through the logs, it seems the seedNodes_0 is very special, other nodes(seedNodes_1, seedNodes_2) sends actor message to seedNodes_0 is very fast,
but other nodes(seedNodes_1, seedNodes_2) send actor message to itself(local node) is slower than send actor message to seedNodes_0.

Someone knows the reason?

https://doc.akka.io/docs/akka/current/cluster-usage.html#joining-configured-seed-nodes

…The reason for the special first seed node is to avoid forming separated islands when starting from an empty cluster. It is quickest to start all configured seed nodes at the same time (order doesn’t matter), otherwise it can take up to the configured seed-node-timeout until the nodes can join

The docs go into more detail but, yes, the first seed node, when using manually specified seed nodes, is special during startup. Of course, this is moot when using more sophisticated cluster initialization like Akka Management/K8.

Thank you for your quick reply.

in seedNode_2, when send actor message using actorSelection to other nodes( seedNodes_0, seedNodes_1, seedNodes_2), can we make the speed is almost same?
In my local test, send to the first node (seedNode_0) is more fast than sending to other nodes(seedNode_1, seedNode_2), e.g

Time of seedNode_2->seedNode_0 = 10ms
Time of seedNode_2->seedNode_2 = 200ms
Speed difference is too big.
I don’t understand the strange phenomenon, it seems sending actor message from any nodes to seedNode_0 is more fast, so that’s reason why i said seedNode_0 is very special

If you are using actorSelection, you are effectively bypassing the cluster infrastructure anyway. So this isn’t related to seed nodes, at least not in any direct way.

How are you measuring?

I tested as below

There has one akka cluster including three nodes. e.g

“CONFIG_akka_cluster_seedNodes_0=akka.tcp://scheduler-actor-system@IP_A:7000”,
“CONFIG_akka_cluster_seedNodes_1=akka.tcp://scheduler-actor-system@IP_B:7001”,
“CONFIG_akka_cluster_seedNodes_2=akka.tcp://scheduler-actor-system@IP_C:7002”,

Then, on seedNodes_2, send actor message to seedNodes_0, seedNodes_1, seedNodes_2, e.g.

val path = s"akka.tcp://scheduler-actor-system@{seedNode_IP}:{port}/user/QueueManager"
val actorSelection = context.actorSelection(path)
actorSelection.ask(…)

Above QueueManager is the actorName

In theory, the three nodes are symmetric, but through the logs, it seems the seedNodes_0 is very special, other nodes(seedNodes_1, seedNodes_2) sends actor message to seedNodes_0’s actor is very fast, but other nodes(seedNodes_1, seedNodes_2) send actor message to itself(local node) is slower than send actor message to seedNodes_0.

I meant, how are you timing the differences in time? Because one possibility was just that you have a server with a drifting system clock. Or that you have some kind of testing warmup difference.

However, your code does illustrate the point. First, I’m assuming that you are properly changing the path with each test. In the example you gave you say @{seedNode_IP}, but since are sending the message directly, it should IP_A, IP_B, IP_C etc, and then 7000, 7001, 7002 respectively.

Second, the code illustrates that you are making the connection directly. It’s not going through the cluster infrastructure at all: you are manually specifying a TCP IP and port. So if the communication between B and A is faster than your communication between B and C, then it’s either an issue with your testing methodology or your network stack. If all else fails, I’d test ping times, because this is definitely not related to being the first seed node.

David

Thanks, i worte a very simple actor to test

class TestManager (implicit logging: Logging)  extends Actor {
  override def receive: Receive = {
    case createQueue: CreateQueue =>
      logging.info(this, s"---------------received hello request, action:${createQueue.fqn}")
      sender ! "hello-back"
  }
}

object TestManager {
  val actorName = "testManager"

  def props()(
    implicit logging: Logging): Props = {
    Props(new TestManager())
  }
}

The result showes that the three seed nodes are symmetric, which means seedNode_N sends actor message to seedNode_1 , seedNode_2 , seedNode_3’s time is almost same.

well, the akka cluster has no problem, it seems it is our code’s problem, so i will look into it more carefully.

Thank you again!