I am currently working on our logging configuration for our application and found out that the frameworks play currently uses, slf4j (API) and Logback (implementation) have not been updated since 2019, with quite a lot of beta releases stalled since then. Especially lambda support is still sitting in beta and thus is implemented on top in play.Logger. This makes it impossible for underlying frameworks for example to defer processing of the lambda to a separate thread etc.
I wonder, are there any plans to move to another logging framework that is still in active development and that supports lambdas, such as log4j?
However, before you do that, do take a look at the wider Logback ecosystem. Most of the logback activity is outside of Logback, i.e.GitHub - logfellow/logstash-logback-encoder: Logback JSON encoder and appenders has regular releases, and I have a Scala logging API called Blindsight which is built on top of it using structured logging concepts to add a DSL and type class support:
And it does have lambda functionality and other fun things like compile time statement interpolation.
This makes it impossible for underlying frameworks for example to defer processing of the lambda to a separate thread etc.
From a CPU perspective, it’s actually faster to do it in the same thread – the work involved in assembling a logging event is minimal. Using a lambda is actually slower as it causes an object instantiation and causes more memory allocation pressure on the backend. I wrote about this in a couple of blog posts:
Thanks for the answer, it is good to know that there is stuff happening around SLF4J and Logback!
SLF4J has to versions in beta. They have been in beta for two years. And SLF4J is still missing a stable API with lambdas. That gets me worried.
Yes, processing a lambda in a separate thread is more CPU intensive. But it can increase responsitivity and also in high load situations the log processing can be dropped as Logback’s async appender does, but not for the log string creation.
For avoiding the log string creation when the line would not be written, wrappers can be used, such as the play Wrapper that provides lambda methods. However it would be nicer if that were in the SLF4J API itself so that it could also be used from play independent libraries.
In general, a project that has had beta versions for two years (which is true for both SLF4J and Logback) gets me worried. There seems to have been a need to develop something but then there weren’t the resources or the need to finish it. Do you have any idea as to why this has happened? My guess was that people have moved to LOG4J and thus people are not willing to spend resources on SLF4J/Logback anymore. Is that true?
No, not at all. I don’t remember the URL, but the last academic paper I read on logging libraries had something like 80% of all libraries using SLF4J and something like 90% of SLF4J users on logback. Apache Commons and Log4J 2 made up the rest of it. Sources here and here, although I know I’ve seen more numbers elsewhere.
In general, the reason why SLF4J and Logback development stopped is because it reached a point where most people were happy with what it did. There were no “lack of resources” but instead a lack of pain. For reference Log4J 2 itself hasn’t seen serious API changes since 2012, and where there are changes it’s upgrades and adding new layouts etc. If you look at the changelog you’ll see most of it: https://logging.apache.org/log4j/2.x/changes-report.html
Log4J 2 I think was intended to be an advancement over Logback, but many of its advancements (LMAX based async loggers, structured logging) have been matched by logstash-logback-appender over time. Likewise, some of the advancements like steady state zero-garbage logging are still unfinished since 2016. There are still some things that Log4J does (using Instant instead of millis, Nested Diagnostic Context) that are better than Logback, but there is not a significant performance difference between the two.
Yes, processing a lambda in a separate thread is more CPU intensive. But it can increase responsitivity and also in high load situations the log processing can be dropped as Logback’s async appender does, but not for the log string creation.
If you’re concerned about log string creation, you’re better off with parameterized string templates. Conditional logging can either evaluate a lambda or a block, but a block is one less memory allocation when called, and an allocation still has a floor of 20ns.
But my point is mostly that in modern JVMs, it doesn’t matter what logging framework you use. The real work of logging is done in the async appender and encoding into JSON using Jackson, which is unreasonably effective.
So logging is not going to be a runtime bottleneck in modern JVMs. Here are my benchmarks for JDK 14. The work done in JDK 16 is even better, with GC max pause times of 0.5ms. If latency or throughput is a concern, then either you’re in high-frequency trading land, or you’re logging enough to stress out the JVM – in which case the thing you have to worry about are the costs of storing and indexing those logs.
In general, I think the bottleneck in logging frameworks is not in performance but in flexibility and observability. It’s not great that MDC is a thread-local, for example, or that there’s no way to target an individual logger as being “debug for a particular request.” That’s most of what Blindsight is about.