Hello,
sorry if this is a very basic question; i tried reading docs on Akka Streams site but the more i read the more confused I am with the alternatives to choose.
I think the question is simple: let’s say I’ve a very simple flow:
src -> map(fn) -> snk
type of elements is not relevant for the purpose of this question.
Assume I’m running this graph in two nodes (N1 and N2) and that services are stateless, running Akka Streams with Alpakka Http-Client (and Http-Server, for the webhooks I’ll explain below).
Let’s say the function fn on node N1 calls an external async service (i.e. a REST API which returns immediately with 202, to later return the real payload through a webhook). The fn function and the external REST API are designed to be stateless, in such a way that I have all the information to avoid persisting the state of the graph: when the webhook is called I should receive the context needed to know the stage to pick and the data to complete the future, thus not requiring to receive the callback in the same node where the call was originated. The question is how to handle the callback if it comes back to node N2.
I’ve a very naïve pipeline implementation that solves this very easy: upon reception of the callback I check the contextual information from the request, instantiate a new pipeline, seek the stage based on this contextual info and then “resume” the workflow by executing this intermediate stage. However I’m interested in Akka Streams for all its stream processing capabilities (i do expect a high load: an infinite source of data with high throughput requirements).
It seems to me like this problem could be solved with StreamRefs in conjunction with Akka Cluster and AsyncCallback, however I’m afraid to overcomplicate things, thus the question. Any advice, pointers or related material would be much appreciated.
Regards