Cluster creation- what is the correct shade plugin to resolve substitutions?

Akka Akka Cluster
1

I am learning by trying to create a very simple cluster. I am running into reference.conf substitution problems at runtime.

Can anyone please show me a pom.xml used with clusters which shows the correct way to use the shade plugin to resolve substitutions?

This is with what I’ve got:

---- My application.conf-

akka
{
  actor
  {
    provider = "cluster"
  }
  remote
  {
    netty.tcp
    {
      hostname = "127.0.0.1"
      port = 0
    }
    artery
    {
      enabled = on
      canonical.hostname = "127.0.0.1"
      canonical.port = 0
    }
  }

  cluster
  {
    seed-nodes =
        ["akka://ClusterSystem@127.0.0.1:2551",
         "akka://ClusterSystem@127.0.0.1:2552"]

    # auto downing is NOT safe for production deployments.
    # you may want to use it during development, read more about it in the docs.
    auto-down-unreachable-after = 10s
  }
}

Enable metrics extension in akka-cluster-metrics.
akka.extensions=["akka.cluster.metrics.ClusterMetricsExtension"]

---- My code-

File configFile = new File("my-path-to/target/application.conf");
assert configFile.exists();
Config parsedConfig = ConfigFactory.parseFile(configFile);
assert parsedConfig != null;
parsedConfig.resolve();
assert parsedConfig.isResolved();
this.actorSystem = ActorSystem.create("my-actor-system", parsedConfig);

---- This last line causes the following-

Exception in thread "main" com.typesafe.config.ConfigException$UnresolvedSubstitution: reference.conf @ jar:file:/Users/soler/.m2/repository/com/typesafe/akka/akka-remote_2.12/2.5.13/akka-remote_2.12-2.5.13.jar!/reference.conf: 879: Could not resolve substitution to a value: ${akka.stream.materializer}
at com.typesafe.config.impl.ConfigReference.resolveSubstitutions(ConfigReference.java:111)
...

---- The shade plugin in my pom.xml-

I combed the forums and tried several shade plugins without success. The most recent is the following shade plugin below.

  <plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-shade-plugin</artifactId>
    <version>3.1.1</version>
      <executions>
        <execution>
          <configuration>
            <transformers>
              <transformer implementation="org.apache.maven.plugins.shade.resource.AppendingTransformer">
                <resource>reference.conf</resource>
              </transformer>
            </transformers>
            <relocations>
              <relocation>
                <pattern>akka.remote</pattern>
                <shadedPattern>shaded.akka.remote</shadedPattern>
              </relocation>
            </relocations>
          </configuration>
        </execution>
      </executions>
  </plugin>
2

How are you packaging and starting your application?

Did you see https://doc.akka.io/docs/akka/current/general/configuration.html#when-using-jarjar-onejar-assembly-or-any-jar-bundler ?

3

Thank you so much, Arnout, for your help.

I am not trying to do anything remotely complicated like packaging the application. I am a newbie and my issue has got to be much simpler than that.

I have backed off a bit to try figure out the basic problem. Now I have simply copied the example distributed in the akka-samples-cluster-java from the cluster usage section of the docs Akka Cluster Sample with Java and placed it in my filesystem. Now application.conf and the other .conf files that came with the example are all in the $CLASSPATH and I made no changes to either SimpleClusterApp.java or SimpleClusterListener.java or to any of the .conf files.

Maven compiled it fine. I added the minimum set of jars to the classpath until I got no more ‘class not found’ runtime exceptions. I set the Java system property -Dconfig.trace=loads when launching so I can verify that application.conf is loaded (along with various other reference.conf). But I still get com.typesafe.config.ConfigException$UnresolvedSubstitution.

Config output + stack trace follows: --------------------

Loading config from a String akka.remote.netty.tcp.port=2551
akka.remote.artery.canonical.port=2551
Loading config from properties {awt.toolkit=sun.lwawt.macosx.LWCToolkit, java.specification.version=9, sun.cpu.isalist=, sun.jnu.encoding=UTF-8, sun.arch.data.model=64, java.vendor.url=http://java.oracle.com/, sun.boot.library.path=/Library/Java/JavaVirtualMachines/jdk-9.0.4.jdk/Contents/Home/lib, sun.java.command=com.gsoler.akkaClusterLab.SimpleClusterApp, jdk.debug=release, java.specification.vendor=Oracle Corporation, java.home=/Library/Java/JavaVirtualMachines/jdk-9.0.4.jdk/Contents/Home, file.separator=/, java.vm.compressedOopsMode=Zero based, line.separator=
, java.specification.name=Java Platform API Specification, java.vm.specification.vendor=Oracle Corporation, sun.management.compiler=HotSpot 64-Bit Tiered Compilers, java.runtime.version=9.0.4+11, user.name=soler, file.encoding=UTF-8, config.trace=loads, java.io.tmpdir=/var/folders/mf/lspn4g2j59ld7l99m1hd109h0000gn/T/, java.version=9.0.4, java.vm.specification.name=Java Virtual Machine Specification, java.awt.printerjob=sun.lwawt.macosx.CPrinterJob, sun.os.patch.level=unknown, java.library.path=/Users/soler/Library/Java/Extensions:/Library/Java/Extensions:/Network/Library/Java/Extensions:/System/Library/Java/Extensions:/usr/lib/java:., java.vendor=Oracle Corporation, sun.io.unicode.encoding=UnicodeBig, socksNonProxyHosts=local|*.local|169.254/16|*.169.254/16, gopherProxySet=false, file.encoding.pkg=sun.io, java.class.path=/Users/soler/projects/MavenProjects/mavenbasedir/target/classes:/Users/soler/.m2/repository/com/typesafe/config/1.3.2/config-1.3.2.jar:/Users/soler/.m2/repository/com/typesafe/akka/akka-actor_2.12/2.5.11/akka-actor_2.12-2.5.11.jar:/Users/soler/.m2/repository/org/scala-lang/scala-library/2.12.4/scala-library-2.12.4.jar:/Users/soler/.m2/repository/com/typesafe/akka/akka-protobuf_2.12/2.5.13/akka-protobuf_2.12-2.5.13.jar:/Users/soler/.m2/repository/com/typesafe/akka/akka-cluster_2.12/2.5.13/akka-cluster_2.12-2.5.13.jar:/Users/soler/.m2/repository/com/typesafe/akka/akka-cluster-tools_2.12/2.5.13/akka-cluster-tools_2.12-2.5.13.jar:/Users/soler/.m2/repository/com/typesafe/akka/akka-cluster-metrics_2.12/2.5.13:/Users/soler/.m2/repository/com/typesafe/akka/akka-remote_2.12/2.5.13/akka-remote_2.12-2.5.13.jar, java.vm.vendor=Oracle Corporation, user.timezone=, os.name=Mac OS X, java.vm.specification.version=9, sun.java.launcher=SUN_STANDARD, user.country=US, http.nonProxyHosts=local|*.local|169.254/16|*.169.254/16, sun.cpu.endian=little, user.home=/Users/soler, user.language=en, java.awt.graphicsenv=sun.awt.CGraphicsEnvironment, ftp.nonProxyHosts=local|*.local|169.254/16|*.169.254/16, path.separator=:, os.version=10.13.5, java.runtime.name=Java(TM) SE Runtime Environment, java.vm.name=Java HotSpot(TM) 64-Bit Server VM, java.vendor.url.bug=http://bugreport.java.com/bugreport/, user.dir=/Users/soler/projects/MavenProjects/akka-cluster-lab, os.arch=x86_64, java.vm.info=mixed mode, java.vm.version=9.0.4+11, java.class.version=53.0}
Loading config from resource 'application.conf' URL file:/Users/soler/projects/MavenProjects/akka-cluster-lab/target/classes/application.conf from class loader jdk.internal.loader.ClassLoaders$AppClassLoader@4f8e5cde
Loading config from a URL: file:/Users/soler/projects/MavenProjects/akka-cluster-lab/target/classes/application.conf
URL sets Content-Type: 'content/unknown'
'content/unknown' isn't a known content type
Loading config from class loader jdk.internal.loader.ClassLoaders$AppClassLoader@4f8e5cde but there were no resources called application.json
exception loading application.json: java.io.IOException: resource not found on classpath: application.json
Loading config from class loader jdk.internal.loader.ClassLoaders$AppClassLoader@4f8e5cde but there were no resources called application.properties
exception loading application.properties: java.io.IOException: resource not found on classpath: application.properties
Loading config from resource 'reference.conf' URL jar:file:/Users/soler/.m2/repository/com/typesafe/akka/akka-actor_2.12/2.5.11/akka-actor_2.12-2.5.11.jar!/reference.conf from class loader jdk.internal.loader.ClassLoaders$AppClassLoader@4f8e5cde
Loading config from a URL: jar:file:/Users/soler/.m2/repository/com/typesafe/akka/akka-actor_2.12/2.5.11/akka-actor_2.12-2.5.11.jar!/reference.conf
URL sets Content-Type: 'content/unknown'
'content/unknown' isn't a known content type
Looking for 'version.conf' relative to ParseableResourceURL(jar:file:/Users/soler/.m2/repository/com/typesafe/akka/akka-actor_2.12/2.5.11/akka-actor_2.12-2.5.11.jar!/reference.conf)
Looking for 'version.json' relative to ParseableResourceURL(jar:file:/Users/soler/.m2/repository/com/typesafe/akka/akka-actor_2.12/2.5.11/akka-actor_2.12-2.5.11.jar!/reference.conf)
Looking for 'version.properties' relative to ParseableResourceURL(jar:file:/Users/soler/.m2/repository/com/typesafe/akka/akka-actor_2.12/2.5.11/akka-actor_2.12-2.5.11.jar!/reference.conf)
Loading config from resource 'version.conf' URL jar:file:/Users/soler/.m2/repository/com/typesafe/akka/akka-actor_2.12/2.5.11/akka-actor_2.12-2.5.11.jar!/version.conf from class loader jdk.internal.loader.ClassLoaders$AppClassLoader@4f8e5cde
Loading config from a URL: jar:file:/Users/soler/.m2/repository/com/typesafe/akka/akka-actor_2.12/2.5.11/akka-actor_2.12-2.5.11.jar!/version.conf
URL sets Content-Type: 'content/unknown'
'content/unknown' isn't a known content type
Loading config from class loader jdk.internal.loader.ClassLoaders$AppClassLoader@4f8e5cde but there were no resources called version.json
exception loading version.json: java.io.IOException: resource not found on classpath: version.json
Loading config from class loader jdk.internal.loader.ClassLoaders$AppClassLoader@4f8e5cde but there were no resources called version.properties
exception loading version.properties: java.io.IOException: resource not found on classpath: version.properties
Loading config from resource 'reference.conf' URL jar:file:/Users/soler/.m2/repository/com/typesafe/akka/akka-cluster_2.12/2.5.13/akka-cluster_2.12-2.5.13.jar!/reference.conf from class loader jdk.internal.loader.ClassLoaders$AppClassLoader@4f8e5cde
Loading config from a URL: jar:file:/Users/soler/.m2/repository/com/typesafe/akka/akka-cluster_2.12/2.5.13/akka-cluster_2.12-2.5.13.jar!/reference.conf
URL sets Content-Type: 'content/unknown'
'content/unknown' isn't a known content type
Loading config from resource 'reference.conf' URL jar:file:/Users/soler/.m2/repository/com/typesafe/akka/akka-cluster-tools_2.12/2.5.13/akka-cluster-tools_2.12-2.5.13.jar!/reference.conf from class loader jdk.internal.loader.ClassLoaders$AppClassLoader@4f8e5cde
Loading config from a URL: jar:file:/Users/soler/.m2/repository/com/typesafe/akka/akka-cluster-tools_2.12/2.5.13/akka-cluster-tools_2.12-2.5.13.jar!/reference.conf
URL sets Content-Type: 'content/unknown'
'content/unknown' isn't a known content type
Loading config from resource 'reference.conf' URL jar:file:/Users/soler/.m2/repository/com/typesafe/akka/akka-remote_2.12/2.5.13/akka-remote_2.12-2.5.13.jar!/reference.conf from class loader jdk.internal.loader.ClassLoaders$AppClassLoader@4f8e5cde
Loading config from a URL: jar:file:/Users/soler/.m2/repository/com/typesafe/akka/akka-remote_2.12/2.5.13/akka-remote_2.12-2.5.13.jar!/reference.conf
URL sets Content-Type: 'content/unknown'
'content/unknown' isn't a known content type
Exception in thread "main" com.typesafe.config.ConfigException$UnresolvedSubstitution: reference.conf @ jar:file:/Users/soler/.m2/repository/com/typesafe/akka/akka-remote_2.12/2.5.13/akka-remote_2.12-2.5.13.jar!/reference.conf: 879: Could not resolve substitution to a value: ${akka.stream.materializer}
	at com.typesafe.config.impl.ConfigReference.resolveSubstitutions(ConfigReference.java:111)
	at com.typesafe.config.impl.ResolveContext.realResolve(ResolveContext.java:179)
	at com.typesafe.config.impl.ResolveContext.resolve(ResolveContext.java:142)
	at com.typesafe.config.impl.SimpleConfigObject$ResolveModifier.modifyChildMayThrow(SimpleConfigObject.java:379)
	at com.typesafe.config.impl.SimpleConfigObject.modifyMayThrow(SimpleConfigObject.java:312)
	at com.typesafe.config.impl.SimpleConfigObject.resolveSubstitutions(SimpleConfigObject.java:398)
	at com.typesafe.config.impl.ResolveContext.realResolve(ResolveContext.java:179)
	at com.typesafe.config.impl.ResolveContext.resolve(ResolveContext.java:142)
	at com.typesafe.config.impl.SimpleConfigObject$ResolveModifier.modifyChildMayThrow(SimpleConfigObject.java:379)
	at com.typesafe.config.impl.SimpleConfigObject.modifyMayThrow(SimpleConfigObject.java:312)
	at com.typesafe.config.impl.SimpleConfigObject.resolveSubstitutions(SimpleConfigObject.java:398)
	at com.typesafe.config.impl.ResolveContext.realResolve(ResolveContext.java:179)
	at com.typesafe.config.impl.ResolveContext.resolve(ResolveContext.java:142)
	at com.typesafe.config.impl.SimpleConfigObject$ResolveModifier.modifyChildMayThrow(SimpleConfigObject.java:379)
	at com.typesafe.config.impl.SimpleConfigObject.modifyMayThrow(SimpleConfigObject.java:312)
	at com.typesafe.config.impl.SimpleConfigObject.resolveSubstitutions(SimpleConfigObject.java:398)
	at com.typesafe.config.impl.ResolveContext.realResolve(ResolveContext.java:179)
	at com.typesafe.config.impl.ResolveContext.resolve(ResolveContext.java:142)
	at com.typesafe.config.impl.SimpleConfigObject$ResolveModifier.modifyChildMayThrow(SimpleConfigObject.java:379)
	at com.typesafe.config.impl.SimpleConfigObject.modifyMayThrow(SimpleConfigObject.java:312)
	at com.typesafe.config.impl.SimpleConfigObject.resolveSubstitutions(SimpleConfigObject.java:398)
	at com.typesafe.config.impl.ResolveContext.realResolve(ResolveContext.java:179)
	at com.typesafe.config.impl.ResolveContext.resolve(ResolveContext.java:142)
	at com.typesafe.config.impl.SimpleConfigObject$ResolveModifier.modifyChildMayThrow(SimpleConfigObject.java:379)
	at com.typesafe.config.impl.SimpleConfigObject.modifyMayThrow(SimpleConfigObject.java:312)
	at com.typesafe.config.impl.SimpleConfigObject.resolveSubstitutions(SimpleConfigObject.java:398)
	at com.typesafe.config.impl.ResolveContext.realResolve(ResolveContext.java:179)
	at com.typesafe.config.impl.ResolveContext.resolve(ResolveContext.java:142)
	at com.typesafe.config.impl.ResolveContext.resolve(ResolveContext.java:231)
	at com.typesafe.config.impl.SimpleConfig.resolveWith(SimpleConfig.java:78)
	at com.typesafe.config.impl.SimpleConfig.resolve(SimpleConfig.java:68)
	at com.typesafe.config.impl.SimpleConfig.resolve(SimpleConfig.java:63)
	at com.typesafe.config.impl.SimpleConfig.resolve(SimpleConfig.java:41)
	at com.typesafe.config.impl.ConfigImpl$1.call(ConfigImpl.java:371)
	at com.typesafe.config.impl.ConfigImpl$1.call(ConfigImpl.java:364)
	at com.typesafe.config.impl.ConfigImpl$LoaderCache.getOrElseUpdate(ConfigImpl.java:65)
	at com.typesafe.config.impl.ConfigImpl.computeCachedConfig(ConfigImpl.java:92)
	at com.typesafe.config.impl.ConfigImpl.defaultReference(ConfigImpl.java:364)
	at com.typesafe.config.ConfigFactory.defaultReference(ConfigFactory.java:367)
	at com.typesafe.config.ConfigFactory.load(ConfigFactory.java:213)
	at com.typesafe.config.ConfigFactory.load(ConfigFactory.java:182)
	at com.typesafe.config.ConfigFactory$1.call(ConfigFactory.java:259)
	at com.typesafe.config.ConfigFactory$1.call(ConfigFactory.java:256)
	at com.typesafe.config.impl.ConfigImpl$LoaderCache.getOrElseUpdate(ConfigImpl.java:65)
	at com.typesafe.config.impl.ConfigImpl.computeCachedConfig(ConfigImpl.java:92)
	at com.typesafe.config.ConfigFactory.load(ConfigFactory.java:256)
	at com.typesafe.config.ConfigFactory.load(ConfigFactory.java:232)
	at com.gsoler.akkaClusterLab.SimpleClusterApp.startup(SimpleClusterApp.java:24)
	at com.gsoler.akkaClusterLab.SimpleClusterApp.main(SimpleClusterApp.java:13)
4

I’d guess you are still missing the akka-streams artifact on your class-path given that the akka.stream.materializer is missing. In general I wouldn’t recommend the “copy artifacts until there is no more class not found exceptions” strategy but rely on the class path the build tool puts together when you have listed all the dependencies.

5

You are correct -it was a missing class at runtime!

I would have expected a java.lang.NoClassDefFoundError and didn’t think a missing class would end up being reported as a config UnresolvedSubstitution exception.

BTW - good advice about relying only on the maven compile classpath at runtime.

Thank you Johan for your help!

6

In this case it is in fact not a missing class, but a missing reference.conf from the streams package, which another reference.conf in akka-remote is expecting to be on the class path (because of Artery, the new remoting implementation being built on top of Akka Streams).

The Typesafe Config library loads all reference.conf files from the class paths and then does variable substitution, if a variable is missing it fails like this so that it does not go unnoticed.

7

Thank you for the clarification.
What is the correct course of action then? add a jar with the missing reference.conf (as I did) or extract the missing reference.conf from the jar and add it by itself to the classpath?

8

It sounds like a bad gamble to guess that only some parts of a dependency is needed unless you are completely sure about that, so adding the entire jar is the way to go.

9

Makes sense.
Thank you all for your help.