Can't join cluster after upgrade to 2.6 (Artery is explicitly turned off)

Hi guys

I’m upgrading akka from 2.5 to 2.6 and I’m having problems with the new 2.6 nodes joining the existing 2.5 cluster. The cluster is using the kubernetes-api for service discovery, and it seems that the node is finding the cluster nodes:

{"akkaAddress":"akka.tcp://fooo@","akkaContactPoints":",,,","sourceThread":"","akkaSource":"akka://foo/system/bootstrapCoordinator","sourceActorSystem":"foo","akkaTimestamp":"10:17:21.731UTC","@timestamp":"2020-08-17T10:17:21.732+00:00","logger_name":"","thread_name":"","level":"INFO","message":"Located service members based on: [Lookup(foo,None,Some(tcp))]: [ResolvedTarget(,None,Some(/, ResolvedTarget(,None,Some(/, ResolvedTarget(,None,Some(/, ResolvedTarget(,None,Some(/], filtered to [,,,]"}
but it fails to join the cluster:

{"akkaAddress":"akka.tcp://foo@","sourceThread":"","akkaSource":"HttpClusterBootstrapRoutes(akka://foo)","sourceActorSystem":"foo","akkaTimestamp":"10:17:22.515UTC","@timestamp":"2020-08-17T10:17:22.515+00:00","logger_name":"","thread_name":"","level":"INFO","message":"Bootstrap request from Contact Point returning 0 seed-nodes []"}

Couldn't join seed nodes after [60] attempts, will try again. seed-nodes=[akka.tcp://foo@, akka.tcp://foo@, akka.tcp://foo@]

On the other nodes Im receiving a serializer exception:

Serializer not defined for message with serializer id [6] and manifest []. Transient association error (association remains live). Unknown manifest [IJ
if I perform a full cluster restart it works, but I should be able to migrate from 2.5 to 2.6 without a full restart right? Artery has been explicitly turned off. Any advice?

The manifest “IJ” reveals that this is related to the serializer of cluster messages, which was changed in Akka 2.6.2 and 2.6.3.
You have to update to 2.6.2 first to have compatible serializer with Akka 2.5.x. Then you can do another rollout to latest 2.6.8. I’ll add a note in the migration guide about this.

Great, thanks!