I’m dealing with the issue of read-side latency and UI responsivness (already briefly discusses in a previous topic). In order to achieve the latter, we need a quickly updated read-side view, even if it is not fully consistent. However, this seems difficult to achieve in a CQRS architecture as we cannot avoid a latency of several seconds between write commands and read-side updates. At least in Lagom, from what I read and tried, even by tuning the parameters it seems difficult to go below a 2 seconds latency.
In the context of Lagom, I see mainly two solutions:
Use the PersistenEntity state as a read-side model
… and for example, put it into the command reply to provide client with the updated view/state. It has several limitations (single-entity view and single kind of view) and it somehow breaks the CQRS pattern by using the write-model as a read-model.
Implement some client-specific predictive view update
For example, when receving a command reply, use it (along with the sent command) to update the current client view, or alternatively propagates events in near rela-time to clients through pubsub and use them to update client view. This solution is quite general and efficient. However, this requires to write situation-specific client code to update the specific client views after each command. Somehow, we need to duplicate part of the view updating code in the client and we have to take care of doing it consistently on both side. More generally, it requires to have part of read-side view management logics on the client side while it should preferably be completely managed on the server side (at least to my opinion)
I used both solutions in different situations and while they work well, I am clearly not satisfied with these becaus of the drawbacks mentioned above (lack of generality & anti-pattern for 1 and view management code duplication for 2) So, if someone has other ideas about how to approach that problem, I would be very glad to hear about it
Towards a more generic solution
On my side, I am thinking about how to implement a kind of generic version of solution 2. The global idea idea would be to have a read-side that do not only listen to events but also to command replies in order to update its content. Hence, we would have some sort of (non-blocking) command proxy so that we could forward any valid command reply to the view in order to do some optimistic update of the view. Then, when the view receive the actual events, it could validate (or invalidate) the updates done based on command replies.
In practice, one approach would be to maintain two copies the view state (consistent and latest) as well as a list of applied (command,reply) pairs. Everytime a new event arrives, it is applied to the consistent view (as in usual read-side). Then, the event is compared with the first element of the applied commands list. If they correspond, the element is simply removed from the list and if they don’t correponds, the latest view is recomputed by re-applying the list of applied commands on the consistent view (after filtering the one corresponding to the current event) .
In general terms, the goal is to have the most up-to-date view besides the eventually consistent view, taking care of not introducing an overhead on the command processing (which could potentially still be used without going through the “reactive view proxy”)
Basically, there would be two requirements:
There should be some way to find the correspondence between a specific event and a specific (command,reply) pair. Ideally, the event offset would seem to be the best way to have such a correspondence but in Lagom it is not possible to have it when constructing the command reply (but maybe it could be possible in principle?!?) For example, we could add an identifier to all events and passing it in command reply.
The same read-side mutation must be defined both for an event and for a (command,reply) pair. Typically, I would imagine a third type ViewUpdate and some (implicit) functions to convert an Event to a ViewUpdate and to convert an (command,reply) pair to a ViewUpdate. Then effective read-side updates are done using ViewUpdate values
From my initial thinking, there are several small issues that need to be adressed (handling multiple events cases) but the main problem I see is about handling efficiently the two view versions in a databases. We need two have two copies of the same database and regularly restore one copy (the latest version) to the state of the other (the consistent version). Alternatively, we should be able to rollback some of the latest changes. For SQL-based, I don’t think there are such features out of the box. However, this could be accomplished by using some kind of automatic genral audit history based on triggers so that we could able able to revert the last DB updates.
Of course, such a solution would add constraints on the definitions of the corresponding commands, events and read view, as well as some overhead on read-side querying and memory usage. But, what I like is that it would potentially provide a reactive read-side view without having to care on the client side about optimistic updates of the view. More generally, this could considerably simplify application code (and make it more consistent)
Another solution I was briefly exploring would be to do that at the query level. Roughly, the idea would be to have specific “reactive query” on the read-side for which the clients get notified about the updates (through WS). But then there are questions on how to accomplish this in a generic fashion and without too much overhead.
I would be happy to hear any comment on this proposed solution. In particular,
- do you think such a pattern is coherent with the general CQRS approach ?
- do you think it could work in practice (and thus significantly reduce the read-side latency) ?
- do you see any potential problem that could arise with such a solution ?
- do you any other idea to avoid the client to manage the view updates ?
- any other comment…
Thanks for taking the time to read!