Google Pub/Sub Actions Scala API

Three problems that we’ve encountered with the Scala pub/sub API…

  1. When consuming from one topic and publishing to another, it seems that we can only do a 1:1 per element consumed. We’d like to do like to be able to consume one element, and potentially produce one or more new elements to a new topic. The API doesn’t appear to allow this. Our work around was to emit “repeated” Protobuf elements.
  2. When consuming from one topic we wanted to be able to fan out to one or more other topics based upon business rules. The API doesn’t appear to support this either. Our workaround was to create separate consumer services, each consuming the same topic, and then filtering based upon business rules for the corresponding destination topic.
  3. Once you integrate with pub/sub, there’s no way to see that the topics are indeed available.

A handler that takes a single request in, and a streaming return, should let you emit multiple or no elements for every incoming element from a topic:

  rpc Consume(ElementIn) returns (stream ElementOut) {
    option (akkaserverless.method).eventing.in = {
      topic:  "from-topic"
    };
    option (akkaserverless.method).eventing.out = {
      topic: "to-topic"
    };
  }

It will only allow a single one destination topic though, so you’d need one handler per topic you want to publish events to. I think you could possibly use the forward effect to do some shared processing of an incoming event and then conditionally forward the processed request to different handlers that each publishes to a different topic.

If the topics does not exist, or you lack access topic permission, when deploying, the deployment is meant to fail with an error in your log saying which topic is missing, but I think there may be a bug there, I’ll look into it.

great recommendation Johan! I was under the impression that streaming wasn’t supported by Akka Serverless, maybe it’s just unsupported on the gRPC endpoint?
I’ll play around with this approach. Thanks.

Streaming out should be supported in most places, both gRPC endpoints and when used for eventing, streaming in works for regular gRPC endpoints but a bit unsure if it works (and if it does, what it would mean for a topic subscriber).