How to create Graph at runtime?

Akka Akka Streams & Alpakka
1

Hello, everybody!
I need to build akka streams graph at runtime. I have some kind of configuration file (blueprint) which describes the topology. So, it is not known at compilation time. But as soon the graph is created it doesn’t change. In this sense it is static. I can use MergeHub, PartitionHub etc. But they allow to build a pure dynamic graph. This is not exactly what I need. Is there any other option?

Thank you,
Andrey

2

I wrote an application you describbed. The biggest problem that this will not be typesafe. If you write a wrapper, and wrap every type to a (TypeName, Any) like datastructure, you can build anything in runtime. (All of your stages will be working with this type.) BUT you will lost type safety so you will need to handle typemismatches…

3

Yes, I understand it. What DSL did you use to create the graph? Could you give an example?
Thank you,
Andrey

4

It was a commercial prodoct, I helped to build to a company (so I have some kind of NDA). The “forntend” and “database” DSL was their own, I just build the runtime graph from a list of what connect to what kind of thigs. Basicaly I wrote a big graph builder from nodes and edges. You will need to do the same if you want something like a frontend pipeline editor to akka streams interpreter.

5

I don’t have much more to share beyond what @tg44 has already mentioned. If the types of your records are known ahead of time then building a typesafe static graph and running it based on information only available at runtime should be achievable. I would imagine using the Graph DSL might be more intuitive while building the graph because you can use simple binary operators to build out that complexity, but you could probably do it with the operator API as well. Assuming your blueprint doesn’t change dynamically at runtime then you could construct the graph and run it without a problem.

As you may already know, the dynamic stream operators (MergeHub, BroadcastHub, etc.) are useful when your consumer and producers themselves aren’t known until runtime. A classic example would be implementing a daemon of some kind where you want clients to be able to join, catch up, and depart without unnecessary stream materializations per client connection, or stream shutdowns/cancellations when clients disconnect. They also provide a way of merging these transient producers (or consumers) together.