To get right to the question: What is the best practice to bridge communication from Akka actors to Python worker instances? (or any other language, for that matter)
On a concrete level, I need use Python for AI neural network training/execution, but I’d love to use Akka with all its benefits for “everything else”, that is, for actual reliable business logic. So the aim is to have Akka instruct Python to train neural networks, and I also need to be able to request a single computation based on real-time data from python at times.
While I can conceive solutions myself, the reason why I am asking is because I’d like to find the most seamless bridge between the two (Akka/Python). In the theoretical ideal case Python would be so well integrated that adding new Python functionality that relies on data from the Actor System is almost as simple as adding another actor.
Lastly, thanks for welcoming me into the community. I’m very amazed by the beauty of Scala and Akka, and have spend my last weeks studying both intensely in parallel to my employment as software developer.
@zeromem no, sorry, don’t have a concrete solution for you, never actually got to this (yet). One thing I was considering though was to use a message-broker (Apache Kafka?) to request and communicate results between Akka and Python.
I don’t know why I haven’t thought of this earlier: Create an actor that starts the Python script as a Process. Then communicate with the Python script via stdin/stdout - the actor manages the communication on top of the process.
Didn’t occur to me that it’s fine to have a local process, opposed to having everything in the network. There can still be multiple instances of the Python actor on multiple machines.
Actually I use a pipe under /dev/shm to solve this problem. Akka gRPC is complex in my case, I just want my Java process (actor) can invoke Python function.
My idea looks similar to what you say,
In my implementation, I have a Python Agent Worker, it is created by Java actor (1 Java actor has 1 Python agent worker if you need). And the python agent is responsible for invoking user’s Python function.
Java actor communicate with Python agent through a pipe in /dev/shm pipe in /dev/shm.
Using this pipe will be more efficient than stdin/stdout.
And if you want to call python function from an remote java actor, just send an PythonFunctionRequest{moduleName, functionName, params} to the remote Java actor, and the remote actor will route this request to the Python agent.
We use gRPC from Akka actors to work with Python interfaced ML models. It is especially useful when leveraging Protobuf all through the platform. And again leveraging a message protocol that is used by Akka internally as well. Works great and makes the platform extensible/polygot and future proof.
Thanks, I think now I have heard gRPC often enough that I’m converted. One question though - do you have to take special considerations in regard to performance when streaming large amounts of data? Cheers
Definitely keep in mind the - source -> flow -> sink template. We have to apply the same mechanisms - internally to manage arriving elements. Since on one side you may not be using a jvm (we have akkastreams etc) you have to be mindful of the tools used to process - with very little or no blocking of outgoing/incoming elements. And definitely have back pressure semantics/protocol meshed in with the flow. The amount of data will never be an issue - if the above aspects are in your design. gRPC is the singular way to go given the explosion of polygot tools that we need in today’s platforms.
