Akka http log levels for production

I feel that Akka HTTP is quite chatty when it comes to logging and want to share my production-based experience to see if we could/should make it less so.

I’ve been running Akka HTTP in production now for well over a year. My HTTP endpoints are exposed to the outside world and don’t have a proxy in front. I realise that Akka HTTP is a fairly low-level abstraction, and perhaps I should have a hardened proxy, but as this particular service only has one instance right now, I wanted to keep the infrastructure as simple as possible.

Here are some of the many, many messages that are filling up my logs:

WARN  ActorSystemImpl - Illegal request, responding with status '400 Bad Request': 
Illegal HTTP message start
WARN  ActorSystemImpl - Illegal request, responding with status '400 Bad Request': 
Unsupported HTTP method: The HTTP method started with 0x16 rather than any known HTTP method. Perhaps this was an HTTPS request sent to an HTTP endpoint?
WARN  ActorSystemImpl - Illegal request, responding with status '400 Bad Request': 
Unsupported HTTP method: HTTP method too long (started with '238ll|'|'|SGFjS'). Increase `akka.http.server.parsing.max-method-length` to support HTTP methods with more characters.
WARN  ActorSystemImpl - Illegal request, responding with status '400 Bad Request': 
Cannot establish effective URI of request to `/`, request has a relative URI and is missing a `Host` header: consider setting `akka.http.server.default-host-header`

That’s just a quick sample. Frankly, I don’t care about any of these as an operator. Would there be an appetite to review all log messages of Akka HTTP and relegate most to DEBUG?

Hi Christopher,

previously, we decided that we would log server-caused errors with ERROR level and client-caused errors with WARN level. It’s not uncommon for webservers to log those kind of incidents because any of those weird requests could be a potential attack attempt (at least that’s what we’ve seen a lot on public servers). Usually, you have those “error.log” files which might contain similar information.

But maybe you are right that putting those out in the regular log with those levels might be overly noisy. Have you tried tuning those messages down by filtering on the logger level?

Johannes

Thanks Johannes. I’ve not applied a logging filter, although I realise that I can. Personally, I think that client-caused messages should be logged at DEBUG. I’m certainly interested in the wider opinion on this though.

2 Likes

Agreed. I’d call this an INFO statement at best, because this isn’t a problem with the ActorSystem or Akka HTTP – it’s an invalid request. You would expect to see WARN level statements on non-error but problematic conditions in the system itself, like lost database connections, reaching max-connection threshold, etc.

2 Likes

It depends on context: when providing an ‘internal’ service, logging those warnings can be helpful. Indeed for an endpoint that receives external traffic it does not make sense. Does setting setting akka.http.server.parsing.error-logging-verbosity to off suppress those warnings? If not, perhaps it should?

I agree that context is important here. As stated earlier, I don’t front the http traffic with a proxy presently; but I probably will end up doing that.

I don’t have verbosity switched off. Controlling these messages with a verbosity setting would make sense if that’s not already the case. Further, having verbosity off by default would also feel right.

For me the log levels seem appropriate, but I think that the logger (ActorSystemImpl) should be more specific. Then it would be easy to configure the logging backend accordingly.

Thanks, Heiko. That’s a good point. We are indeed logging to many things from system.log.

I do wonder in this case, if the log messages are not actionable then what utility are they serving? Right now, my logs are just filling up. Perhaps I should just use a proxy though.