Implementing the Saga / Compensating Transaction pattern in Lagom


(Adam Gray) #1

Hi Lagomers

Following my point on the Gitter channel, I’m exploring Lagom and really like what I see so far. To get my head around how it would fit a real-world use case I’m implementing a textbook saga-type use-case - transferring an amount of money between two accounts but it’s not immediately obvious to me where in the Lagom model the saga orchestrator should sit- in the Service implementation? In a PersistentEntity? In a ReadSideProcessor? I can see arguments for and against each.

For starters here’s a very rough SagaManager which orchestrates the saga steps and the appropriate compensatory actions at the Service level as a sequence of composed command futures:

Then, in the ServiceImpl:

@Override
  public ServiceCall<CreatePaymentRequest, Done> createPayment() {
    return request -> {
      SagaManager sagaManager = new SagaManager(
          Arrays.asList(
              new SagaStep<Done>(
                  "Reserve src",
                  ok -> accountRef(request.getSrcAccountId())
                      .ask(new ReserveCash("", request.getAmount())),
                  ok -> accountRef(request.getSrcAccountId())
                      .ask(new ReleaseCash(request.getAmount()))),
              new SagaStep<Done>(
                  "Credit dest",
                  ok -> accountRef(request.getDestAccountId())
                      .ask(new Credit(request.getAmount())),
                  ok -> accountRef(request.getDestAccountId())
                      .ask(new Debit(request.getAmount()))),
              new SagaStep<Done>(
                  "Release src reserved",
                  ok -> accountRef(request.getSrcAccountId())
                      .ask(new ReleaseCash(request.getAmount())),
                  ok -> accountRef(request.getSrcAccountId())
                      .ask(new ReserveCash(request.getAmount()))),
              new SagaStep<Done>(
                  "Debit src",
                  ok -> accountRef(request.getSrcAccountId())
                      .ask(new Debit(request.getAmount())),
                  ok -> accountRef(request.getSrcAccountId())
                      .ask(new Credit( request.getAmount())))
              ));

      return sagaManager.begin();

There are several obvious problems with this implementation, not least that there is no durable “saga log” - state is only stored in-memory. It also

  • depends on the rather ugly pattern of a command replying null to signal failure, as I can’t see a way of elegantly handling a real failure (e.g. ctx.invalidCommand()) without killing the whole process
  • does not allow steps to pass output to each other

I’d be really interested to hear any approaches that others have taken to this, and how this one could be improved.

Adam


(Alan Klikic) #2

@acgray
PersistantEntity should be used as saga coordinator:

  1. entity state represents current saga step
  2. entity behavior represents the saga step flow (what commands are applicable)
  3. events and readside processor (also topic producer) are used for performing saga step external actions (calling external services, storage actions, publishing to kafka,…)
  4. commands are used to start saga and to commuicate saga step external actions result back to the entity
    a) readside processor sending command after external action is done
    b) topic cosumer by consuming message representing external action result - in case external action was trigger with topic producer

In use case of transferring money (simple example):
MoneyTransferEntity
Behaviours:

  1. in process of taking money from source account
  2. source account take failed (saga end)
  3. in process of putting money on destination account
  4. destination account put failed
  5. source account take rollbacked (saga end)
  6. money transferred (saga end)

Commands:

  1. Initate money transfer
  2. set source account take failed
  3. set source account take successful
  4. set destination account put failed
  5. set destination account put successful
  6. set source account take rollback successful

Events:

  1. source account take initiated
  2. source account take failed
  3. destination account put initiated
  4. destination account take failed
  5. source account take rollbacked
  6. money transferred

Triggering actions based on events can be done in these ways:

  1. event processor event handler triggering other sevice call(s). Result is sending command. If trigger fails but it is not a permanent fail you can throw exception and readside processor will retry. If it is a permanent fail or success you send command.
  2. topic producer publishing event to kafka, consumer consuming message and sending command. Here you maybe also need some kind of timeout trigger if consumes messae is not received in defined time.

Hope this helps.


(Damon Rolfs) #3

Thanks for the extensive explanation, which is helpful. The gap I see is an area I think may be the most prevalent scenario: a saga integrating with an external legacy REST service that returns info via its response; i.e., it doesn’t post events to your event journal nor does it publish events to Kafka. So perhaps there’s a 4.c) given your list above.

One way may be to implement a proxy lagom service to wrap that external service and integrate in accordance to what you’ve described. I can see that may be valuable for circumstances with broader integration, but it’s heavy-handed.

A lighter approach may be to call the service from the saga entity (wrapped in a Future) and pipe the response into a corresponding saga-command transformation sent back to the saga.

I appreciate your thoughts and any pointer to reference or example implementation that demonstrates best practices here.

Thanks, Damon


(Alan Klikic) #4

@dmrolfs legacy app call is also an external action (so it can be triggered via #1 readside processor). As you said, you can wrap it in Lagom service or you can call it directly from readside processor. Depends on the interface of a legacy app.
There are no examples that I know of but will try to do it myself when i find time.
Implementations that I have are to tailored for our solution and are not good showcase.


(Nico Serfontein) #5

An example of the process manager/saga pattern using Lagom would be very helpful indeed.

Some ideas for this showcase:

  • Where to do external calls (think legacy/blocking)
  • At-least-once with idempotence
  • Retry with backoff (and failure after giving up)
  • How to handle compensation steps

If something like this already exists, please let me know.

Thanks,
Nico