Hello everyone,
I’m encountering an issue while working with Akka Stream and I’m hoping someone can assist me in resolving it.
Here’s the context: I have two systems, let’s refer to them as SystemA and SystemB, both deployed on a Kubernetes (k8s) cluster. SystemA is deployed with two replicas, namely SystemA-1 and SystemA-2, while SystemB has only one instance.
SystemA exposes two APIs: API-1, which receives data from other systems, and API-2, which returns a source to other systems. When SystemB starts up, it randomly selects one of the SystemA replicas and consumes API-2. Essentially, API-2 acts as a pipeline between SystemA and SystemB. Currently, SystemA receives data from API-1, and I’ve implemented a Source.queue in SystemA to offer the incoming data from API-1 to API-2. Consequently, SystemB receives data from SystemA, but the issue arises because SystemA has multiple replicas. Due to the load balancing strategy, different replicas receive different data (e.g., data 1, 2, 3, 4). Data 1 and 3 are sent to SystemA-1, while data 2 and 4 are sent to SystemA-2. However, since SystemB only connects to a randomly chosen replica, it only receives data from either SystemA-1 (data 1, 3) or SystemA-2 (data 2, 4). What I aim for is to enable SystemB to receive all data (1, 2, 3, 4).
I suspect the issue lies in my usage of Source.queue for handling data delivery. After researching, I came across MergeHub as a potential solution to this problem. However, I’m unsure how to implement the code. If anyone has a solution or suggestions, I would greatly appreciate your assistance. Thank you.