Roadmap for sbt 1.4?

Congratulations for getting sbt 1.3.0 out!

Would it be possible to have more information about the roadmap for the next version of sbt?

In particular, I’d be curious to know what are the plans about sbt server. The documentation says that this is experimental and that LSP is supported. Do you plan to support BSP?

1 Like

timing

We were fortunate to get lots of help from the contributors for sbt 1.3.0, and were able to make some big changes. For sbt 1.4.0, maybe we should try to ship in 100 ~ 200d. (I’ve said that about sbt 1.3.0 too, but we should aim for it nonetheless)

specifics

One of the things I would be looking into is Zinc performance in large corporate codebase.

BSP support is something I’ve had on a short list of feature enhancement. (Related contribution would be welcome in this area)

Another nice-to-have that I’ve been hacking on-and-off is using build.properties file more. For example, giter8.version should be specified in there so the template author can specify the Giter8 version (much like how sbt works). Another idea that’s been brought up is allowing build authors to specify JVM flags in there.

If anyone has Christmas wishes, I’d be happy to get feedback in this thread, and maybe contributors can get ideas.

3 Likes

Why do those have to wait for 1.4.0?

  1. Every fresh code has bugs, so feature enhancements need to go through RC cycle. (Patch releases come out without RCs)
  2. Extending the life of patch 1.2.x or 1.3.x is one of major causes of delaying the feature release.

Thanks Eugene for your response!

Another thing that comes to my mind: I remember there has been some work to improve “matrix configurations” support. What are your thoughts about it?

The experiments in sbt-projectmatrix is shaping up. It needs some more tweaking, but this might be a good candidate to merge into mothership if people want the feature.

3 Likes

+1000 to that, but at this point I think the best way to achieve this would be to simply bundle the bloop sbt plugin with sbt, like 1.3 did with the coursier sbt plugin. This would also bring all the other improvements bloop brings (classloader handling, build pipelining, …) for free.

1 Like

Currently, the bloop sbt plugin does not much apart from generating a bloop configuration that you can then use with bloop. So, bundling the bloop sbt plugin within sbt would not help, in its current state, if I understood correctly.

So, I guess what we need for a working integration is to make sbt delegate compilation to the bloop server, and delegate the “sbt server” to the bloop server. Maybe @jvican can confirm or fix this assumption?

That being said, I’ve seen that mill chose to not use bloop to support BSP (see lihaoyi/mill#664), although there is a mill integration in bloop, which does provide BSP for free. I’m not familiar enough with either bloop or mill to fully understand the reasons for this, but it seems that the mill integration in bloop can not take advantage of some features of mill (I guess this is by design, because bloop is designed to be build tool agnostic). I’m wondering whether the same kind of limitations would apply if we were trying to get BSP support (and “all the improvements bloop brings for free”) by relying on the sbt bloop integration?

1 Like

Given Maven Central is so… central I think it’s high time to bring jar signing (whether it’s sbt-pgp or sbt-gpg’s approach) and sonatype staging repo operations (close/release repo, etc) into sbt.

3 Likes

The BSP implementation for Mill doesn’t use Bloop because it would have been a larger change to Mill to integrate with Bloop, and these are somewhat orthogonal changes. The drawback of using Bloop as BSP server is that it doesn’t know about the task structure of the build tool, so if the build does anything besides compile, such as code generation, this will not be supported as part of the build.

That said, BSP support in sbt would enable a more robust integration with IntelliJ, so I’m personally very interested in this.

2 Likes

Two things that come to mind for me for 1.4 are:

  • Overhaul the launcher. The current version uses ivy and is stuck on an old version of scala and sbt 0.13 without much of an upgrade path. It also has a brittle proguard configuration that I was unable to easily upgrade. Ideally, we could rework the launcher so that it bootstraps itself in java and once it has identified the scala version of the target sbt version, it can run other initialization code using that version of scala. This could conceivably also speed up start up time since we are effectively loading parts of the scala standard library twice. A fork of https://github.com/coursier/sbt-launcher might be a good place to start.
  • Make sbt -client default or at least good enough that it could be the default. There are some features, most notably ~ that do not work with the thin client. The current thin client implementation that is bundled in sbt is also quite slow to launch (I think it takes at least a half second on my machine). For supported platforms, we could conceivably provide a native implementation ala https://github.com/cb372/sbt-client and use a java base implementation for other platforms. The java implementation could be sped up by changing the implementation to pure java with minimal libraries to reduce classloading overhead.
2 Likes

No, integrating with Bloop would take significantly less lines than 1705 lines in Mill https://github.com/lihaoyi/mill/pull/664 and its implementation would be trivial.

The build server protocol doesn’t allow clients to know about the task structure either and that’s on purpose. I’ve never understood what’s the motivation of allowing to introspect the whole task graph of a build tool from the editor, it’s most of the time useless aside from the core tasks: compiling, testing and running. There’s no reasonable use case that requires exposing source generation to the build client IMO.

BSP support in sbt would enable a more robust integration with IntelliJ,

More robust? I don’t think so. You can say it would be more elegant from a design point of view but for all purposes the BSP server would behave ~ the same whether it’s implemented inside sbt or bloop.

In fact, a BSP implementation in sbt would be more detrimental than one in Bloop: users cannot have an sbt shell running shell at the same time IntelliJ is sending compile requests to get diagnostics. That is a big deal for users.

Instead of writing a new launcher in Java, why not use coursier’s coursier’s launch capabilities (or bootstrap, bootstrap --standalone, bootstrap --native)?

coursier/sbt-launcher (aka “csbt”) is basically a re-implementation of sbt/launcher, except without all the self-bootstrapping/classloader/proguarding shenanigans.

2 Likes

No, integrating with Bloop would take significantly less lines than 1705 lines in Mill https://github.com/lihaoyi/mill/pull/664 and its implementation would be trivial.

I didn’t see a trival way to do this within the current design of Mill, but I’m happy to be convinced otherwise.

The build server protocol doesn’t allow clients to know about the task structure either and that’s on purpose. I’ve never understood what’s the motivation of allowing to introspect the whole task graph of a build tool from the editor, it’s most of the time useless aside from the core tasks: compiling, testing and running. There’s no reasonable use case that requires exposing source generation to the build client IMO.

Misunderstanding here. The build client doesn’t need to know about source generation or other parts of the task graph. But the build tool will run it as a dependency of the compile request. Bloop will not. So if a build does anything besides compilation as part of a regular build, this will not be replicated by Bloop.

More robust? I don’t think so. You can say it would be more elegant from a design point of view but for all purposes the BSP server would behave ~ the same whether it’s implemented inside sbt or bloop.

More robust in that the communication happens over a well-defined protocol, rather than a hacky shell integration. As to the behavior, see above.

Yeah. I think the coursier launcher may be a good starting point. At the moment, it is slower than the sbt launcher though. I find it takes about a second longer to boot a project but I haven’t investigated why.

The reason I think a java bootstrap is a good idea is because the test interface classloader needs to come above the scala library classloader in the hierarchy. That’s difficult to implement if the launcher entry point is implemented in scala. The java part could be very small though.

I prefer Jason’s idea of using Coursier or whatever to resolve the JARs in bash script, but you directly launch Main from bash without either sbt launcher or csbt’s sbt launcher reimplementation. That kills two birds with one stone.

One caveat is that we would need to reimplement repository override feature in the mothership, but that’s doable.

There’s a whole bunch more caveats to that. And for what gain?

I already spent time with Jason looking at Coursier’s bootstrapping and he was happy with how hygienic it was when launching an app.

From https://twitter.com/eed3si9n/status/1176203373365878784

Change the meaning of ++ <sv> to a prefix search of crossScalaVersions and picking the first match. Credits to https://github.com/coursier/coursier/blob/master/project/ScalaVersionPlugin.scala for the inspiration.

The motivation is to use build.sbt for the single source of truth for full Scala version, and only use 2.12 or 2.13 in .travis.yml.

The gain is simpler runtime.

  • If we don’t have to load either Coursier or Ivy that should speed up the load.
  • The same goes for not having to JIT older version of Scala library.
  • One fewer layer for ClassLoader trick.

You should check if those are issues when using Coursier.