[idea] Add publishStaging task to sbt for creating bundles for Sonatype upload

I’d like to propose adding publishStaging (and publishStagingSigned) task to sbt, in order to publish Maven artifacts to a specific folder in the local machine.

Background

sbt-sonatype 3.6 has a feature to upload thousands of files as a bundle to minimize the network overhead of publishing tasks:

  • This bundling scheme significantly improves the upload performance to Sonatype because it requires only a single network connection.
  • Recently the latency of Sonatype API is increasing. If we use publishSigned directly to the API, the latency of the API accumulates and releasing projects tasks can take several hours. This is because sbt needs to perform operations in publishSigned task sequentially: build an artifact file (e.g., .jar, .src) -> apply GPG signature -> upload files to Sonatype. This process will be repeated for each artifact file, so even if the latency of Sonatype API is negligible for small projects, projects that have thousands of artifacts may take several hours just for finishing publishSigned task.
  • With bundle upload, uploading thousands of files can be finished in several minutes.

Currently, sbt-sonatype requires overriding publishTo setting to specify a folder like target/sonatype-staging/(version)/ to collect all Maven artifacts to a single place. This is because Sonatype API only supports uploading a single folder as a bundle to a staging repository at Sonatype.

If we have publishStaging task (like publishM2) in sbt, we can create a local bundle folder at ease without overriding publishTo setting. sbt-pgp also need to have publishStagingSigned task to create GPG-signed artifacts locally to a specific target folder.

Proposal

Add following sbt tasks to sbt and sbt-pgp:

  • publishStaging

    • Publish Maven artifacts of modules to a specific local folder
    • Basically we need to use a single target folder in the root project (e.g., s"(ThisBuild / target).value/staging/{version.value}"). If necessary, by setting a key like stagingDirectory := s"{target.value}/staging/$(version)", users can create a bundle folder for each sub module.
  • For sbt-pgp:

    • publishStagingSigned
      • GPG-signed version of publishStaging
3 Likes

Since people are used to typing publishSigned, it might be better to create something like publishStageTo key that would override the behavior of publishTo.

The behavior would be:

  1. If the build has publishStageTo set to Some(directory), publish and publishSigned would publish to the specified stage directory.
  2. Otherwise, it would follow publishTo.

Reusing publishTo might work, but we also need to consider the following things:

  • Sonatype is not supporting releasing snapshot version bundles, so for snapshot versions, publishTo should not create any staging folders.
  • In sbt-sonatype 3.6, publishTo := sonatypePublishToBundle.value setting creates a local staging folder for release versions, but for snapshot versions, it uploads artifacts directly to the Sonatype snapshots repository to retain the usual behavior. Although it works, this sonatypePublishToBundle setting is doing a little bit too much; users will have no idea what will happen just by looking this setting key.

From the consistency viewpoint of sbt task names, adding publishStaging looks more natural to me and easier to understand:

  • publish (for using a remote repository)
  • publishLocal (for using the local .ivy2 folder)
  • publishM2 (for using the local .m2 folder)
  • publishStaging (for using a local staging folder)

Actually there is a ticket for that and guys says they will consider including it into 1.4: https://github.com/sbt/sbt/issues/4958