Restart sources

Akka Akka Streams & Alpakka
1

I’m trying to understand the behaviour of using RestartSource.withBackoff.

It looks like one of my sources is restarted more times than the maxRestart count. It looks like this is happening becaues in RestartWithBackoffLogic.maxRestartsReached() has a resetDeadline that can pass in which the restartCount is reset to 0.

This isn’t documented in the documentation, so I’ve been trying to understand what triggers this behaviour. It looks like if I have a minBackof of 1 second, but sometimes the source takes longer than 1 second to fail, the restart count will repeated be reset to 0 and hence the source will be restarted forever.

Does anyone have more insights into this behaviour? If I’m correct about the behaviour, and if so, why does it behave this way and should it be in the API documentation?

2

The behaviour that you describe is the intended one. From the api doc of the RestartSource.withBackoff:

  • @param maxRestarts the amount of restarts is capped to this amount within a time frame of minBackoff. Passing 0 will cause no restarts and a negative number will not cap the amount of restarts.

The maxRestarts parameter is meant to prevent rapid restarts until the exponential backoff gets large enough. So it controls the maximum amount of restarts that can happen in a time frame equal to the one specified in the minBackoff parameter.

1 Like
3

So, it sounds like there’s no way to specify an absolute cap on the max number of restarts, and so it could potentially be restarting forever? It sounds like this is different than the Backoff strategy for actors.

I’m a bit confused why maxRestarts would apply to the number of restarts within a minBackoff interval:

   * @param minBackoff minimum (initial) duration until the child actor will
   *   started again, if it is terminated

It sounds like the source shouldn’t be restarted sooner than minBackoff. So why would we have 1 or more restarts within that time frame?

4

Sorry for a late reply. The resetDeadline is useful on for sources that fail immediately. I can see how it can become useless with sources that take a while to fail. If you think having something that would control the total amount of restarts would be useful to you, please create a ticket in the akka issue tracker where we can discuss it further.

5

I am absolutely stunned at your decision to take an argument called “maxRestarts” and, for no reason I can see, use it for the sub-minBackoff scenario. What on Earth would make you think that an Exponential Backoff wouldn’t need a restart-cap?

2 Likes
6

Trying to understand this. Won’t the RestartSource wait at least minBackoff duration before starting again? So how can there ever be more than one restart in a minBackoff interval?

7

I am using RestartSource, and I was bothered by not understanding this, so I looked into the code. I see the question was asked before by @gowlin, howbeit three years ago.

The previously given wording was not quite accurate in my opinion. The best way I can summaries the behavior is like this: If the time-to-fail since the last time the source was restarted is less than minBackoff, the failure is counted towards maxRestarts. Whenever a failure takes longer than minBackoff since restart, the restart counter, in respect to maxRestarts, is reset.

minBackoff serves a dual purpose as minimum backoff duration, as well as the above. On a critical note, there is no logical relationship between minBackoff and maxRestarts, and neither the naming (nor documentation) makes this non-intuitive behavior clear.

E: Maybe it would have been better to add another parameter resetRestartCounterAfter or similar.