Spatial representation of cross building

sbt

(Eugene Yokota) #1

sbt has scoping axis (subproject, config, and in-task). When I started hacking on sbt, I remember calling scalaVersion key “the axis of evil” because cross building on Scala versions is one of few places where the entire setting values are overturned using ++2.12.6 or +compile command. The situation gets more hairy when there are dependencies between the subproject, or if the test frameworks are missing. In 2014, I experimented with sbt-doge partially to address by inverting the aggregation, but it’s more of a bandaid.

In 2018, there are additional cross building axes platform (consisting of JVM, JS, and Native), sbtVersion, and JDK implementations.

Steps

  1. I have module fooCore, fooApp, and fooPlugin. Let’s say both fooApp and fooPlugin depend on fooCore; fooCore supports 2.10 ~ 2.13.0-M4, JVM and JS; fooApp supports 2.12 JVM; fooPlugin supports 2.10 JVM and 2.12 JVM.
  2. I want to build and test them.

Spacial representation

Instead of mutating the build state with ++2.12.6, the direction we should explore is spreading them out spatially as subprojects. This idea was pioneered by Tobias Schlatter’s addition of CrossProject mechanism to Scala.js plugin. This was later expanded to sbt-crossproject.

lazy val fooCore = crossProject(JSPlatform, JVMPlatform)
  .settings(
    testFrameworks ++= Seq(
        TestFramework("sbttest.framework.DummyFramework"),
        TestFramework("inexistent.Foo", "another.strange.Bar")
    )
  )

lazy val fooCoreJS = fooCore.js
lazy val fooCoreJVM = fooCore.jvm

However, this only addresses the platform cross building, but not any other parameters, like Scala version.

Enter sbt-cross written by Paul Draper in 2015.

lazy val fooApp = (project in file("app")) 
  .dependsOn(fooCore_2_11)
  .settings(scalaVersion := "2.11.8")

lazy val fooPlugin = (project in file("plugin"))
  .dependsOn(fooCore_2_12)
  .settings(scalaVersion := "2.12.1")

lazy val fooCore = project.cross
lazy val fooCore_2_11 = fooCore("2.11.8")
lazy val fooCore_2_12 = fooCore("2.12.1")

I have not used sbt-cross personally, but I think it has a great potential to simplify the cross building. It apparently provides support for cross building on library axis as well.

This notion of build matrix is something Li Haoyi implements in Mill’s Cross Builds as well.

Likely what we want is a combination of sbt-crossproject and sbt-cross, but more on that later.

Source directory layout

  • sbt already implements Scala version specific source directories (for example src/main/scala-2.12/) in addition to the normal src/main/scala directory.
  • The sbt-crossproject CrossType.Full convention of using shared/, jvm/, js/ directories are the defacto standard for Scala.js and Native cross building.

Since it’s more common to do Scala version cross building only on JVM, it likely make sense to keep the src/main/scala/, src/main/scala-2.12/ convention by default. Combining CrossType.Full convention and scala-2.12/ looks like this:

core/
  +- shared/
  |    +- src/
  |         +- main/
  |              +- scala/
  |              +- scala-2.12/
  +- jvm/
  |    +- src/
  |         +- main/
  |              +- scala/
  |              +- scala-2.12/
  +- js/
       +- src/
            +- main/
                 +- scala/
                 +- scala-2.12/

Instead, the following might be more consistent:

core/
  +- src/
       +- main/
            +- scala/
            +- scala-2.12/
            +- scala-jvm/
            +- scala-jvm-2.12/
            +- scala-js/
            +- scala-js-2.12/

Composite project

sbt 1.2.0 adds a notion called composite project originally proposed by Sébastien Doeraene in #3042 and implemented by BennyHill in #4056. It provides a hook to tell sbt about subprojects without explicitly creating lazy val fooCoreJVM.

projectMatrix proposal

Building off of the composite project, I am thinking about projectMatrix DSL:

lazy val fooCore = (projectMatrix in file("core"))
  .settings(
     name := "foo-core"
  )
  .scalaVersions("2.12.6", "2.11.12")
  .jvmPlatform(scalacOptions := ...)

The above will generate fooCoreJVM2_12 and fooCoreJVM2_11 automatically.

I think we mostly need idSuffix and directorySuffix for generalization:

lazy val fooCore = (projectMatrix in file("core"))
  .settings(
     name := "foo-core"
  )
  .scalaVersions("2.12.6", "2.11.12")
  .custom(
    idSuffix = "Dispatch010",
    directorySuffix = "-dispatch0.10",
    scalaVersions = Seq("2.11.12"),
    settings = Seq(libraryDependencies += ...)
  )

Using custom(...) hopefully the plugin ecosystem could extend this to support Scala.JS and Native:

lazy val fooCore = (projectMatrix in file("core"))
  .settings(
     name := "foo-core"
  )
  .scalaVersions("2.12.6", "2.11.12")
  .jsPlatform(scalacOptions := ...) // provided by plugins
  .nativePlatform(scalaVersions = Seq("2.11.12"), scalacOptions := ...)

Let us know what you think. Also, let us know if you want to help implement this thing.


(Eugene Yokota) #2

I have now have a PR for the basic feature of the above - https://github.com/sbt/sbt/pull/4200


(Sébastien Doeraene) #3

I don’t have a strong opinion on the topic. It is probably worth exploring. I believe @olafurpg wanted to do something like that.

One thing to take into account: more projects being reified at the same time means that many times more settings, which means an increased loading time of sbt. I personally don’t care about sbt’s loading time, but I’ve heard many people do …


(Konrad `ktoso` Malawski) #4

Hey Eugene,
this looks fine, but I wanted to ask/include in the discussion how we should go forward with directory names for “jdk version” which some projects will need due to multi-release-jars and java modules from 9 onwards where we need to swap in “not using unsafe” implementations for things etc.

Currently:

            +- scala/
            +- scala-2.12/
            +- scala-jvm/
            +- scala-jvm-2.12/

and via the https://github.com/sbt/sbt-multi-release-jar

 `src/main/scala-jdk9`
 `src/main/java-jdk9`

any thoughts how we should move forward with combining those? e.g.

src/main/scala-jvm-2.12-jdk9

or different?


(Ólafur Páll Geirsson) #5

I am super excited about this idea and I would love to have better built-in support in sbt for cross-building without ++. This change would be a big incentive to upgrade my builds to sbt 1 (scalafmt, scalafix and scalameta are still on 0.13).

In the Scalafix build, we use a hand-rolled variant of sbt-cross. This was necessary to support some weird dependencies we had between projects from 2.10 up to Dotty. However, the build code is tricky to understand and extend so I would love to migrate from it.

I don’t have suggestions for exact implementation details, but some use-cases I have encountered and would like to see covered

  • remove suffix for the “default” cross-build so the id is short and nice. For example, if JVM on 2.12 is the default then those projects can be referenced in the sbt shell as foo/compile instead of fooJVM212/compile
  • share sources between any subset of cross-projects, for example JS and JVM but not Native. Something like scala-jvm-js/, scala-jvm-native/, scala-js-native/ would do.
  • define custom scalaVersions for a specific platform, for example Scala Native supports only 2.11.
  • cross-build against different versions of a library dependency
  • a.dependsOn(b) should work for any composite projects a and b as long as the cross-build for b is a superset of the cross-build for a (ideally with helpful error messages when it’s not!)

(Sébastien Doeraene) #6

Your bullets 1, 3 and 5 are supported by sbt-crossproject, btw ;-)


(Eugene Yokota) #7

Yes. JDK cross building should be part of this (See also cross JDK forking that I’m adding in sbt 1.2).

Since the convention I am proposing is

+- main/
       +- scala-$suffix/
       +- scala-$suffix-2.12/

sbt-multi-release seems ok, but I guess what I am not sure is if the JDK 9/10/11 axis is orthogonal to platform axis JVM/JS/Native. In other words, would the platform cross testing look like:

  • JVM with JDK8, JVM with JDK11, JS
  • JVM with JDK8, JVM with JDK11, JS with JDK8, JS with JDK11

(Eugene Yokota) #8

Thank you!

I’ve been thinking that too, and here’s my proposal for generalizing the concept: https://github.com/sbt/sbt/issues/4210. Once we have that, compositeProjects could automatically generate those ProjectRefs.

That’s something the build user land can probably do easily, I think. Similar demand is splitting pre- and post-2.13 Scala collection change.

Right, the matrix can be sparsely populated. In one of fooCore examples, custom(scalaVersions = Seq("2.11.12")) is doing that to override the default scalaVersions seq.

Yes. That’s what .custom(...) does in the above. These are all going to be subprojects, so you can also add arbitrary settings into the platform-variants.

Yes. The projectMatrix proposal implements that, I think - https://github.com/sbt/sbt/pull/4200


(Julien Richard Foy) #9

I guess this is not possible but I would like to have just one sbt project instead of one per target platform pre Scala version per ….


(Kerr) #10

What if for different Scala version we need a different dependencies?
Seems like that I have to go with the . custom(..)


(Eugene Yokota) #11

I don’t think you’d need .custom(...) for that. You just need to configure your setting as you’d currently do with some if expression on scalaVersion.value.


(Filipe Regadas) #12

Good stuff! however, with this idea how would we build/test for one specific scalaVersion/platform?


(Sébastien Doeraene) #13

fooCoreJVM2_12/test, assuming I understand the question.


(Eugene Yokota) #14

Assuming you’d have multiple matrices like fooCore, fooApp, fooPlugin, etc., and you want to split the work on CI, I think you’d set up a dummy subproject that aggregates whatever the axis that you want to filter in.


(Filipe Regadas) #15

Yup, splitting the work on CI was the case I was thinking.

Having a dummy subproject as you suggested sounds ok and I guess it gives more customization as well. But, is something like sbt <some_version_plat_switch> <command> ... out of the question? would it be more user friendly?


(Dale Wijnand) #16

How do you run the unit tests for one particular target platform? How do you compile the sources for a particular target scala version?

Personally I think this way forward has promise, but the naming is unfortunate. And to be honest, the naming is bad already as it’s more common to refer to these as “modules” than “projects” (thus why sbt sometimes refers to these as “subprojects” or “multi-module builds”… :roll_eyes:).


(Julien Richard Foy) #17

We would use settings to disambiguate between the axes of the matrix. Like we currently do with scalaVersion and crossScalaVersions.


(Dale Wijnand) #18

The way we currently do with scalaVersion is to use a command (+ or ++), which in turn uses session settings to modify the build state before running test.

The idea here is to flatten these variations into n “projects” (again: bad name) so then you can do things like: specify which to aggregate, execute in parallelise, etc.


(Guillaume Martres) #19

I think it’s just a UI problem, instead of having all modules defined at the root, modules should sit in a hierarchy so you can write something like fooCore/JVM/2.12/compile (and fooCore/xxx could alias fooCore/JVM/2.12/xxx)


(Dale Wijnand) #20

More axes? :confused: