Graceful termination on SIGTERM using akka-http


#1

Hello, wondering if anyone would be able to help me with how to handle SIGTERM in akka-http? The current (10.1.3) Akka HTTP docs:
https://doc.akka.io/docs/akka-http/current/server-side/graceful-termination.html
talk about graceful termination, using this code sample:

import akka.actor.ActorSystem
import akka.http.scaladsl.server.Directives._
import akka.http.scaladsl.server.Route
import akka.stream.ActorMaterializer
import scala.concurrent.duration._

implicit val system = ActorSystem()
implicit val dispatcher = system.dispatcher
implicit val materializer = ActorMaterializer()

val routes = get {
  complete("Hello world!")
}

val binding: Future[Http.ServerBinding] =
    Http().bindAndHandle(routes, "127.0.0.1", 8080)

// ...
// once ready to terminate the server, invoke terminate:
val onceAllConnectionsTerminated: Future[Http.HttpTerminated] =
Await.result(binding, 10.seconds)
  .terminate(hardDeadline = 3.seconds)

// once all connections are terminated,
// - you can invoke coordinated shutdown to tear down the rest of the system:
onceAllConnectionsTerminated.flatMap { _ ⇒
  system.terminate()
}

I am wondering where to put the above termination code snippets so that they are invoked on a SIGTERM?

I also found a slightly different way of shutdown from https://github.com/akka/akka-http/issues/1210#issuecomment-338825745, not sure which of these I should be using?

CoordinatedShutdown(system).addTask(
    CoordinatedShutdown.PhaseServiceUnbind, "http_shutdown") { () =>

    bind.flatMap(_.unbind).flatMap { _ =>
      Http().shutdownAllConnectionPools
    }.map { _ =>
      Done
    }
  }

I’ve tried putting this into the shutdown hook:

CoordinatedShutdown(system).addCancellableJvmShutdownHook{
  // once ready to terminate the server, invoke terminate:
  val onceAllConnectionsTerminated: Future[Http.HttpTerminated] =
  Await.result(binding, 10.seconds)
    .terminate(hardDeadline = 3.seconds)

  // once all connections are terminated,
  // - you can invoke coordinated shutdown to tear down the rest of the system:
  onceAllConnectionsTerminated.flatMap { _ ⇒
    system.terminate()
  }
}

But requests in progress are ended immediately upon sending a SIGTERM (kill ), rather than completing.

Note, question also posted here: https://stackoverflow.com/questions/51401024/handle-sigterm-in-akka-http

Many thanks!


(Michael Pisula) #2

Hi there,
I am using something similar to

CoordinatedShutdown(system).addTask(
    CoordinatedShutdown.PhaseServiceUnbind, "http_shutdown") { () =>

    bind.flatMap(_.terminate(hardDeadline = 1.minute)).map { _ =>
      Done
    }
  }

For me the main part was to increase akka.coordinated-shutdown.default-phase-timeout as it took longer to finish the processing of the request than the 5 second default. You can also just increase the timeout for that one phase. I had the following message in my logs:

Coordinated shutdown phase [service-unbind] timed out after 5000 milliseconds

Hope that helps.

Cheers,
Michael


(Michael Pisula) #3

@akkateam:
If this is a generally accepted solution would it make sense to put it into the docs until Akka HTTP is integrated into coordinated shutdown per default? Would happily create a PR if you agree.
Cheers,
Michael


#4

Thanks @michaelpisula, I can confirm this works for me.


(Patrik Nordwall) #5

@michaelpisula please add it to https://github.com/akka/akka-http/issues/1210 if it’s not already there. Help with updating the documentation is also welcome.