Speed at which an event goes through the microservices


(franz) #1

Hi,

Let’s say i have a simple event that basically just gets passed around from one microservice to another. If that event goes through 5 microservices, how long will it usually take to go through all 5 microservices?

Microservice 1 – produces EventA
Microservice 2 – Subscribes to Microservice 1. Receives EventA and Publishes EventA
Microservice 3 – Subscribes to Microservice 2. Receives EventA and Publishes EventA
Microservice 4 – Subscribes to Microservice 3. Receives EventA and Publishes EventA
Microservice 5 – Subscribes to Microservice 4. Receives EventA and Stores

What’s the fastest way to get that event go through those microservices and how long would it take? Will it take less than a second? …or will it take more than 5 seconds…or will it take 5 minutes?

Thanks!


Lagom event latency
(Alan Klikic) #2

Hi @franz,

There are multiple factors affecting processing time.
But to make things simple and assuming publish is done from the event stream, from my experience, the most time influencing part was wrongly configured cassandra query journal configuration.

This is a configuration that I use, and was suggested by Lagom core team:

cassandra-query-journal.eventual-consistency-delay = 200ms
cassandra-query-journal.delayed-event-timeout = 30s
cassandra-journal.pubsub-minimum-interval = 200ms

For simple cases, with this configuration, processing time per service (subscribe, persist event, publish) will be less then 1 second.

Br,
Alan


(franz) #3

Thanks @aklikic. we’ll try that out!

Where can I find more information about those configuration? I cant seem to find them from the docs


(Alan Klikic) #4

@franz
reference.conf


(franz) #5

Thanks @aklikic we tried that and there were some improvements, but still not within the performance that we want (i.e. goes through all microservices in less than a second).

I’ll post in awhile some more additional findings that we got.


(franz) #6

We tried adding some logging in one of our microservices, to learn more where in the microservice are we getting a performance issue.

  1. The first graph (left most) was the time difference between LOG Marker #1 and LOG Marker #2 (i.e. Microservice #1 - Command creation to command handler)
  2. The 2nd graph was the time difference between LOG Marker #2 and LOG Marker #3 (i.e. Microservice #1 - Command handler to ctx.thenPersist(…))
  3. The 3nd graph was the time difference between LOG Marker #3 and LOG Marker #4 (i.e. Microservice #1 - ctx.thenPersist(…) to event publication)
  4. The 4th graph was the time difference between LOG Marker #4 and LOG Marker #5 (i.e Microservice #2 event publication to Microservice #2 event subscription)

From the graphs above, the 3rd and 4th graph are the slowest. Any thoughts as to why that is and as to how to improve them?

Thanks!

Appendix. Code Snippets

public class Microservice1ExternalWebsocketListener {
    // connects to an external service via websocket and creates a command per message
    private void sendCommand(ExternalWebsocketMessage websocketMessage) {
        PersistentEntityRef<MyCommand> entityRef = persistentEntityRegistry.refFor(MyEntity.class,
        websocketMessage.getIdentifier());

        MyUuid uuid = MyUuid.genThenLog(uuidLogger, SERVICE_NAME); // LOG Marker #1
        MyCommand.Update updateCmd = new MyCommand.Update(uuid, websocketMessage);

        entityRef.ask(updateCmd);
    }
}
public class MyEntity {
...
    behaviorBuilder.setCommandHandler(MyCommand.Update.class, (cmd, ctx) -> {
          logger.debug("Got a new command {}", cmd);

          cmd.getUuid().log(uuidLogger, "COMMANDRECEIVED"); // LOG Marker #2
          return ctx.thenPersist(
              new MyEventImpl.Updated(cmd.getUuid(), cmd.getPayload()),
              evt -> {
                MyUuid uuid = new MyUuid("COMMANDRECEIVED", cmd.getUuid().getUuid());
                uuid.log(uuidLogger, "EVENTPERSISTED");  // LOG Marker #3
                ctx.reply(Done.getInstance());

                MyEventImpl.Updated updatedEvt = (MyEventImpl.Updated) evt;
                pubSubEntityUpdated.publish(EntityToUIWebsocketMessage.translate(updatedEvt.getUuid(), updatedEvt.getPayload()));
              }
          );
        }
    );
...
}
public class Microservice1MyServiceImpl {
  @Override
  public Topic<MyEvent> myEventTopic() {
    return TopicProducer.taggedStreamWithOffset(MyEventImpl.SHARDING_TAG.allTags(), (tag, offset) ->
        persistentEntityRegistry.eventStream(tag, offset)
            .map(this::convertEventImpl2Api));
  }

  private Pair<MyEvent, Offset> convertEventImpl2Api(Pair<MyEventImpl, Offset> eventOffsetPair) {
    MyEvent eventToPublish;

    if (eventOffsetPair.first() instanceof MyEventImpl.Updated) {
      MyEventImpl.Updated updated =
          (MyEventImpl.Updated) eventOffsetPair.first();

      eventToPublish = new MyEvent.Updated(
          updated.getUuid(),
          updated.getPayload()
      );
    } else {
      throw new IllegalArgumentException("Unknown event: " + eventOffsetPair.first());
    }

    MyUuid uuid = new MyUuid("EVENTPERSISTED", eventToPublish.getUuid().getUuid());
    uuid.log(uuidLogger, "EVENTCONVERTED");   // LOG Marker #4
    return Pair.create(eventToPublish, eventOffsetPair.second());
  }
}
public class Microservice2Subscriber {
...
  private void subscribeToMicroservice1MyService() {
    microservice1MyService.myEventTopic()
        .subscribe().atLeastOnce(myFlow());
  }

  private Flow<MyEvent, Done, NotUsed> myFlow() {
    return Flow.<MyEvent>create().mapAsync(1, myEvent -> {
      MyUuid uuid = new MyUuid("EVENTCONVERTED", myEvent.getUuid().getUuid());
      uuid.log(uuidLogger, SERVICE_NAME); // LOG Marker #5

      ...
    });
  }
...
}

(Alan Klikic) #7

Hi @franz,

Graph #3 for me looks ok (it is a periodical and pubsub triggered query of a journal and publishing to kafka).
Graph #4 looks to slow. Taking in mind that time difference is almost linear I would check kafka subscriber configuration. I do not think publishing should be a problem.
In my expirience i did not notice delays between publishing and subscribing. I’m using default configuration.


(Azhar uddin) #8

Hi, Thanks for the information.

Regards
Azhar
Microservices Developer