Akka StreamRefs with sharded workers

Hi,

Has anyone any experience with the new StreamRefs? I’m looking to use them in a cluster where a job offers a SourceRef to sharded workers and just wondering what happens if a worker node gets re-balanced, will the stream recover? Is there a backoff & retry?

The documentation: https://doc.akka.io/docs/akka/2.5/stream/stream-refs.html mentions the following:

Stream refs are absolutely expected to be sent over Akka remoting to other nodes within a cluster, and as such, complement and do not compete with plain Actor messaging. Actors would usually be used to establish the stream, by means of some initial message saying “I want to offer you many log elements (the stream ref)”, or alternatively in the opposite way “If you need to send me much data, here is the stream ref you can use to do so”.

What happens during failure after the initial offer of a stream ref…?

Thanks!

Hi Graham,

It’s only the StreamRef that can be serialized and sent to other nodes in the cluster. Once it’s materialized you have a local running stream that can’t be relocated. When your sharded actor is re-balanced (stopped) that stream has to end and you have to establish a new stream from another StreamRef from the new location.

Depending on the use case you might want to manage some offset to be able to continue from where you were. That is something you have to add yourself, in the actor message exchange, when retrieving the StreamRef from for a given offset.

Regards,
Patrik

Ah ok I understand now that makes much more sense! Thanks very much for getting back to me Patrik.