Three problems that we’ve encountered with the Scala pub/sub API…
- When consuming from one topic and publishing to another, it seems that we can only do a 1:1 per element consumed. We’d like to do like to be able to consume one element, and potentially produce one or more new elements to a new topic. The API doesn’t appear to allow this. Our work around was to emit “repeated” Protobuf elements.
- When consuming from one topic we wanted to be able to fan out to one or more other topics based upon business rules. The API doesn’t appear to support this either. Our workaround was to create separate consumer services, each consuming the same topic, and then filtering based upon business rules for the corresponding destination topic.
- Once you integrate with pub/sub, there’s no way to see that the topics are indeed available.