1000s of concurrent timed tasks with akka serverless

Hi,

Really been enjoying having a go with akka serverless recently - it’s very refreshing to have a neat programming model where you don’t have to faff about setting up infrastructure for storing state!

Please forgive my ignorance if this has an easy solution as I’m basically brand new to Akka and Akka serveless.

Anyways, I have a use case where I have 100s and maybe 1000s of IoT devices, and I want to create an alarm if no data is received for a set period of time, lets say 10 minutes.

The ways I can think of to do this in non-akka java, would be to:

  1. Use something like ScheduledExecutorService or TimerTask from pure java 11, or perhaps something from rxjava timer or interval, something like Vert.x timer, or Spring’s scheduledTimerTask, and get the timers to run after the 10 minutes, and check whether data has been received.
  2. Constantly iterate over a list of times/instants, checking whether they are less than 10 minutes after the last seen data.
  3. Use a proper library for doing scheduling like quartz, or even more involved, using something like temporal with akka serverless just forming the “services” that you call out to in your workflows.

I realise I could implement one of the above points in Akka serverless, but I’d probably have to do some kind of side-effect thing - return the response and leave the service running, and not sure whether that is intended?

I have even tried the following dumb code, and it seems to work:

ScheduledExecutorService executor;

private void waitUntil(Instant deadline) throws InterruptedException {
    while(Instant.now().isBefore(deadline)){
      synchronized(deadline){
        executor.schedule(
                ()->{
                  System.out.println("Waiting until" + deadline.toString());
                  synchronized(deadline){
                    deadline.notifyAll();
                  }
                },
                Duration.between(Instant.now(),deadline).toMillis(),
                TimeUnit.MILLISECONDS
        );
        deadline.wait();
      }
    }
  }

But it doesn’t feel very “Actor” like, and would seem to be blocking the thread, so not sure how well it scales, and whether if I invoke it for thousands of instances I am gonna have trouble with a single service deployment/project (and whether it is going to be massively expensive in the long run!).

It would seem that what I want in Akka parlance is shown here or something like the akka-quartz-scheduler component. I suppose I could do something with external infrastructure (message busses etc.) but I’d prefer not to (since akka serverless is so neat otherwise!)

Are there easy ways to integrate this stuff into akka serverless? Or, am I barking up the wrong tree, so to speak with akka serverless instances not really being actors and thus expecting to have long running in-memory operations is not the right model or use case?

Cheers,
Harvey.

Hi Harvey! Sorry for the late reply. Too much turkey during the Thanksgiving break!

I love the use case. I’ve been noodling on this need recently, respective of Akka Serverless. I agree that our primary focus has been those “long running in-memory operations” (although note that data is persisted in a DB for value and event sourced entities). But I myself have had some use cases - essentially mirroring the IoT scenario from you - in which I want to have some awareness of time elapsed since last event (or other significant milestone) and then execute logic at specific times or intervals.

I head up Product Management here at Lightbend. @retgits is better equipped to talk about the actual need as observed in the market more generally. I don’t think either one of us is equipped to talk about lower level Akka though.

Hi @HarveyEllis!

Great questions and absolutely love the use case!

Let me start by adding some context on Actors in Akka Serverless. Whenever you deploy a service to Akka Serverless, we deploy a sidecar/proxy that essentially “houses” all the Actors. Whenever Akka Serverless gets a request, we route it to the right proxy and that proxy lets the right Actor respond to it. If the Actor is already in-memory, the Actor forwards the request to your code (and your code receives both the current state and the data of the request). If the Actor isn’t in-memory yet, it’ll be loaded from the database (and follows the rest of the process as above). So there are definitely Actors involved, the user code you write, though, aren’t the actual Actors themselves. I hope that clarifies it a bit, if not, feel free to drop me a DM.

If I understand your use case correctly, you want to get a notification when an entity (IoT device) hasn’t been updated in a specific period of time? Assuming I understood that correctly and depending a little on your implementation, there are a few options you have with Akka Serverless today.

What you could do, is every time an entity gets updated, you also emit an event to build up a View. Your View would in that case be a list of all devices and the last time they were updated. You can define a query that either retrieves all data and loop over it in an Action or you could even specify your query to only return a subset of the data. The Action invoking the View can loop over the results and send notifications. To execute the trigger, you could use something like Google Cloud Scheduler or Temporal (though the latter might be a bit overkill for that).

We’re looking at whether adding something like a scheduler for jobs makes sense for Akka Serverless as a platform, so your use case is definitely interesting to read. If you have any thoughts or want to have a chat, feel free to send a DM!

Hey both,

Thanks for the replies!

No problem - I’d assumed people might be away and doing other things!

I guess I’d say kind of what I’m looking for is to be able to “suspend” an actor and then have it come back again at a certain time in the future (so that, in this case, it can check whether any more data has been received within the time period). I think this is probably something you can do in the model where all the actors are kind of long running processes in memory, I just wanted to know whether there was any mechanism for that in Akka serverless.

At the current time Akka serverless seems to be very good for all the standard use cases basically having some entities and storing information about them, and I think it neatens up event sourcing stuff too - certainly much easier to get started with than Kafka in my experience. These things probably make up somewhere between 70-90% of enterprise use cases I’d imagine. The other stuff, where time is involved - things like workflows, and this use case for example, where you’d want to notify people or perform actions at certain times seem a little bit lacking. And I mean - a tool can’t be all things to all people, so I understand if the solution is to “integrate external infrastructure”, but I guess it would be nice to have things like scheduling, or waiting or some kind of abstraction inside the framework because every little thing you have to integrate is just a little bit more friction (maybe I am a bit too much of a “make everything inside the programming model” purist in this regard!)

Ah right - so your code is not an actor, but do you get 1 actor being spun up for each request that gets received?

I guess what I was trying to get at is - if I block and wait in my code what is the scaling going to be like for 1000s of invocations. I was imagining because actors are very cheap process-wise, and you can have thousands or even hundreds of thousands on a single node, would it be possible to just get the framework to create thousands of actors in a waiting state and then have the framework just automatically wake them up at the right time - saving some effort of effectively having to do all the timings myself.

The reason I mentioned temporal was because I think I could probably implement the timer model in there - as in, I create a workflow for each IoT device, and then those workflows go to sleep until a certain point - very much like the way I described I wanted to use actors above.

Yeah - this was the kind of thing I was thinking I might have to do. The use of a view is a very good idea though - I like that! Perhaps what I am actually looking for is some kind of “reactive view” if you will. Something that collate all the entities where current_time - last_data_received_time > 10, but have that automatically updated and kept in check - basically what you said but without having to do the polling myself. It’s not a hardship really to do the polling, but if there was something like this you would be able to create a scheduling system from it. Might be a bit hard to implement though - having to check entities against certain conditions, I think this is where the long running in-memory representation of actors might be better.

I think some kind of scheduler definitely makes sense to look at - of course there are alternatives that you can use, like pubsub with cloud scheduler, but I’d appeal to completeness of the programming model here.

Also, personally I think it would be (and feel) much neater if you could do fine-grained scheduling though i.e. on the entity or event level. For example, I submit a request to the scheduler with my entity_id, and a continuation/callback location (whatever is in my api_proto for example, or it could just be to callback the method called it) and then when the time is right it returns that. You could do this yourself with the cloud scheduler like you mentioned, but it is not going to be as efficient as if it was implemented in the platform itself I don’t think.

Anyways, thanks both, and I hope this clarifies a few things I said in the OP.

Your code is indeed not an Actor. The entities you create are spun up as Actors and each individual device will be an Actor (they can be passivated when they’re not being used for a period of time and rehydrated when a messages comes in to save on resource usage). When those Actors get requests to act on, they’ll forward that to your code who will handle it and return the result to the Actor.

Thanks very much @retgits - that clears things up!