Lagom to 1.4.10 - java.lang.OutOfMemoryError?


(Joo) #1

Hi all,

After upgrading the Lagom to 1.4.10, our Circle Ci build runs into this OOM issue. I re-tried it many times after increasing the container size but same.

I am not 100% sure if this is caused by the Lagom upgrade, because almost at the same time, I upgraded the SBT from 1.1.5 to 1.2.8, so it could be also why.

I upgraded SBT to be able to have the following options to be included in our build.sbt. I am not sure if that is causing this issue.


lazy val startServiceLocatorCassandraKafka = Command.command("startServiceLocatorCassandraKafka") { state =>
  var newState = Command.process("lagomServiceLocatorStart", state)
  newState = Command.process("lagomKafkaStart", newState)
  newState = Command.process("lagomCassandraStart", newState)
  newState
}

lazy val stopServiceLocatorCassandraKafka = Command.command("stopServiceLocatorCassandraKafka") { state =>
  var newState = Command.process("lagomServiceLocatorStop", state)
  newState = Command.process("lagomKafkaStop", newState)
  newState = Command.process("lagomCassandraStop", newState)
  newState
}

commands ++= Seq(startServiceLocatorCassandraKafka, stopServiceLocatorCassandraKafka)

Full error logs:

[warn] 			callers: com.typesafe.play:play_2.12:2.6.21 ()
[warn] 	com.typesafe.play:play-exceptions
[warn] 		- 2.6.21
[warn] 			status: release
[warn] 			publicationDate: 2019-01-08T09:52:13Z
[warn] 			resolver: sbt-chain
[warn] 			artifactResolver: sbt-chain
[warn] 			evicted: false
[warn] 			homepage: https://playframework.com
[warn] 			extraAttributes: Map(info.apiURL -> https://www.playframework.com/documentation/2.6.21/api/scala/index.html)
[warn] 			isDefault: false
[warn] 			configurations: compile, runtime(*), master(compile), runtime, compile(*), master
[warn] 			licenses: (Apache-2.0,Some(http://www.apache.org/licenses/LICENSE-2.0.html))
[warn] 			callers: com.typesafe.play:build-link:2.6.21 ()
[warn] 	com.lightbend.lagom:lagom-spi_2.12
[warn] 		- 1.4.10
[warn] 			status: release
[warn] 			publicationDate: 2019-01-10T18:19:30Z
[warn] 			resolver: sbt-chain
[warn] 			artifactResolver: sbt-chain
[warn] 			evicted: false
[warn] 			homepage: https://www.lagomframework.com/
[warn] 			isDefault: false
[warn] 			configurations: compile, runtime(*), master(compile), runtime, compile(*), master
[warn] 			licenses: (Apache-2.0,Some(http://www.apache.org/licenses/LICENSE-2.0.html))
[warn] 			callers: com.lightbend.lagom:api-tools_2.12:1.4.10
[error] java.lang.OutOfMemoryError: Java heap space
[error] 	at java.util.Arrays.copyOf(Arrays.java:3332)
[error] 	at java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:124)
[error] 	at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:448)
[error] 	at java.lang.StringBuilder.append(StringBuilder.java:136)
[error] 	at sjsonnew.support.scalajson.unsafe.JsonPrinter.printString(JsonPrinter.scala:65)
[error] 	at sjsonnew.support.scalajson.unsafe.JsonPrinter.printString$(JsonPrinter.scala:58)
[error] 	at sjsonnew.support.scalajson.unsafe.CompactPrinter$.printString(CompactPrinter.scala:54)
[error] 	at sjsonnew.support.scalajson.unsafe.CompactPrinter.$anonfun$printJObject$2(CompactPrinter.scala:40)
[error] 	at sjsonnew.support.scalajson.unsafe.CompactPrinter.$anonfun$printJObject$2$adapted(CompactPrinter.scala:39)
[error] 	at sjsonnew.support.scalajson.unsafe.CompactPrinter$$Lambda$2479/1675164709.apply(Unknown Source)
[error] 	at sjsonnew.support.scalajson.unsafe.JsonPrinter.$anonfun$printArray$1(JsonPrinter.scala:95)
[error] 	at sjsonnew.support.scalajson.unsafe.JsonPrinter.$anonfun$printArray$1$adapted(JsonPrinter.scala:93)
[error] 	at sjsonnew.support.scalajson.unsafe.JsonPrinter$$Lambda$2480/646944597.apply(Unknown Source)
[error] 	at scala.collection.IndexedSeqOptimized.foreach(IndexedSeqOptimized.scala:32)
[error] 	at scala.collection.IndexedSeqOptimized.foreach$(IndexedSeqOptimized.scala:29)
[error] 	at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:194)
[error] 	at sjsonnew.support.scalajson.unsafe.JsonPrinter.printArray(JsonPrinter.scala:93)
[error] 	at sjsonnew.support.scalajson.unsafe.JsonPrinter.printArray$(JsonPrinter.scala:91)
[error] 	at sjsonnew.support.scalajson.unsafe.CompactPrinter$.printArray(CompactPrinter.scala:54)
[error] 	at sjsonnew.support.scalajson.unsafe.CompactPrinter.printJObject(CompactPrinter.scala:39)
[error] 	at sjsonnew.support.scalajson.unsafe.CompactPrinter.printJObject$(CompactPrinter.scala:37)
[error] 	at sjsonnew.support.scalajson.unsafe.CompactPrinter$.printJObject(CompactPrinter.scala:54)
[error] 	at sjsonnew.support.scalajson.unsafe.CompactPrinter.print(CompactPrinter.scala:31)
[error] 	at sjsonnew.support.scalajson.unsafe.CompactPrinter.print$(CompactPrinter.scala:29)
[error] 	at sjsonnew.support.scalajson.unsafe.CompactPrinter$.print(CompactPrinter.scala:54)
[error] 	at sjsonnew.support.scalajson.unsafe.CompactPrinter.$anonfun$printJObject$2(CompactPrinter.scala:42)
[error] 	at sjsonnew.support.scalajson.unsafe.CompactPrinter.$anonfun$printJObject$2$adapted(CompactPrinter.scala:39)
[error] 	at sjsonnew.support.scalajson.unsafe.CompactPrinter$$Lambda$2479/1675164709.apply(Unknown Source)
[error] 	at sjsonnew.support.scalajson.unsafe.JsonPrinter.$anonfun$printArray$1(JsonPrinter.scala:95)
[error] 	at sjsonnew.support.scalajson.unsafe.JsonPrinter.$anonfun$printArray$1$adapted(JsonPrinter.scala:93)
[error] 	at sjsonnew.support.scalajson.unsafe.JsonPrinter$$Lambda$2480/646944597.apply(Unknown Source)
[error] 	at scala.collection.IndexedSeqOptimized.foreach(IndexedSeqOptimized.scala:32)
[info] Compiling 7 Scala sources to /home/circleci/silver-server/redemption-api/target/scala-2.12/classes ...
[error] ## Exception when compiling 21 sources to /home/circleci/silver-server/timeSeries-impl/target/scala-2.12/classes
[error] GC overhead limit exceeded
[error] scala.collection.mutable.ResizableArray.$init$(ResizableArray.scala:32)
[error] scala.collection.mutable.ArrayBuffer.<init>(ArrayBuffer.scala:49)
[error] scala.tools.nsc.classpath.AggregateClassPath.mergeClassesAndSources(AggregateClassPath.scala:97)
[error] scala.tools.nsc.classpath.AggregateClassPath.list(AggregateClassPath.scala:85)
[error] scala.tools.nsc.symtab.SymbolLoaders$PackageLoader.doComplete(SymbolLoaders.scala:271)
[error] scala.tools.nsc.symtab.SymbolLoaders$SymbolLoader.complete(SymbolLoaders.scala:220)
[error] scala.tools.nsc.symtab.SymbolLoaders$SymbolLoader.load(SymbolLoaders.scala:236)
[error] scala.reflect.internal.Symbols$Symbol.$anonfun$typeParams$1(Symbols.scala:1757)
[error] scala.reflect.internal.Symbols$Symbol.completeTypeParams$1(Symbols.scala:1757)
[error] scala.reflect.internal.Symbols$Symbol.typeParams(Symbols.scala:1764)
[error] scala.reflect.internal.Types.isRawIfWithoutArgs(Types.scala:3909)
[error] scala.reflect.internal.Types.isRawIfWithoutArgs$(Types.scala:3909)
[error] scala.reflect.internal.SymbolTable.isRawIfWithoutArgs(SymbolTable.scala:18)
[error] scala.reflect.internal.tpe.TypeMaps$$anon$1.apply(TypeMaps.scala:325)
[error] scala.reflect.internal.tpe.TypeMaps$$anon$1.apply(TypeMaps.scala:322)
[error] scala.reflect.internal.Symbols$Symbol.modifyInfo(Symbols.scala:1563)
[error] scala.reflect.internal.Symbols$Symbol.cookJavaRawInfo(Symbols.scala:1712)
[error] scala.tools.nsc.typechecker.Infer$Inferencer.checkAccessible(Infer.scala:272)
[error] scala.tools.nsc.typechecker.Typers$Typer.makeAccessible(Typers.scala:579)
[error] scala.tools.nsc.typechecker.Typers$Typer.$anonfun$typed1$56(Typers.scala:5007)
[error] scala.tools.nsc.typechecker.Typers$Typer$$Lambda$10739/1676846511.apply(Unknown Source)
[error] scala.tools.nsc.typechecker.Typers$Typer.silent(Typers.scala:700)
[error] scala.tools.nsc.typechecker.Typers$Typer.typedSelect$1(Typers.scala:5007)
[error] scala.tools.nsc.typechecker.Typers$Typer.typedSelectOrSuperCall$1(Typers.scala:5064)
[error] scala.tools.nsc.typechecker.Typers$Typer.typedInAnyMode$1(Typers.scala:5587)
[error] scala.tools.nsc.typechecker.Typers$Typer.typed1(Typers.scala:5603)
[error] scala.tools.nsc.typechecker.Typers$Typer.runTyper$1(Typers.scala:5640)
[error] scala.tools.nsc.typechecker.Typers$Typer.typedInternal(Typers.scala:5672)
[error] scala.tools.nsc.typechecker.Typers$Typer.body$2(Typers.scala:5613)
[error] scala.tools.nsc.typechecker.Typers$Typer.typed(Typers.scala:5618)
[error] scala.tools.nsc.typechecker.Typers$Typer.typedQualifier(Typers.scala:5723)
[error] scala.tools.nsc.typechecker.Typers$Typer.typedQualifier(Typers.scala:5729)
[error]            
[error] java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError: GC overhead limit exceeded
[error] 	at java.util.concurrent.FutureTask.report(FutureTask.java:122)
[error] 	at java.util.concurrent.FutureTask.get(FutureTask.java:192)
[error] 	at sbt.ConcurrentRestrictions$$anon$4.take(ConcurrentRestrictions.scala:213)
[error] 	at sbt.Execute.next$1(Execute.scala:110)
[error] 	at sbt.Execute.processAll(Execute.scala:113)
[error] 	at sbt.Execute.runKeep(Execute.scala:90)
[error] 	at sbt.EvaluateTask$.liftedTree1$1(EvaluateTask.scala:420)
[error] 	at sbt.EvaluateTask$.run$1(EvaluateTask.scala:419)
[error] 	at sbt.EvaluateTask$.runTask(EvaluateTask.scala:438)
[error] 	at sbt.internal.Aggregation$.$anonfun$timedRun$4(Aggregation.scala:99)
[error] 	at sbt.EvaluateTask$.withStreams(EvaluateTask.scala:355)
[error] 	at sbt.internal.Aggregation$.timedRun(Aggregation.scala:97)
[error] 	at sbt.internal.Aggregation$.runTasks(Aggregation.scala:111)
[error] 	at sbt.internal.Aggregation$.$anonfun$applyTasks$1(Aggregation.scala:67)
[error] 	at sbt.Command$.$anonfun$applyEffect$2(Command.scala:137)
[error] 	at sbt.internal.Aggregation$.$anonfun$evaluatingParser$11(Aggregation.scala:212)
[error] 	at sbt.internal.Act$.$anonfun$actParser0$3(Act.scala:435)
[error] 	at sbt.Command$.process(Command.scala:181)
[error] 	at sbt.MainLoop$.processCommand(MainLoop.scala:151)
[error] 	at sbt.MainLoop$.$anonfun$next$2(MainLoop.scala:139)
[error] 	at sbt.State$$anon$1.runCmd$1(State.scala:246)
[error] 	at sbt.State$$anon$1.process(State.scala:250)
[error] 	at sbt.MainLoop$.$anonfun$next$1(MainLoop.scala:139)
[error] 	at sbt.internal.util.ErrorHandling$.wideConvert(ErrorHandling.scala:16)
[error] 	at sbt.MainLoop$.next(MainLoop.scala:139)
[error] 	at sbt.MainLoop$.run(MainLoop.scala:132)
[error] 	at sbt.MainLoop$.$anonfun$runWithNewLog$1(MainLoop.scala:110)
[error] 	at sbt.io.Using.apply(Using.scala:22)
[error] 	at sbt.MainLoop$.runWithNewLog(MainLoop.scala:104)
[error] 	at sbt.MainLoop$.runAndClearLast(MainLoop.scala:59)
[error] 	at sbt.MainLoop$.runLoggedLoop(MainLoop.scala:44)
[error] 	at sbt.MainLoop$.runLogged(MainLoop.scala:35)
[error] 	at sbt.StandardMain$.runManaged(Main.scala:138)
[error] 	at sbt.xMain.run(Main.scala:89)
[error] 	at xsbt.boot.Launch$$anonfun$run$1.apply(Launch.scala:109)
[error] 	at xsbt.boot.Launch$.withContextLoader(Launch.scala:128)
[error] 	at xsbt.boot.Launch$.run(Launch.scala:109)
[error] 	at xsbt.boot.Launch$$anonfun$apply$1.apply(Launch.scala:35)
[error] 	at xsbt.boot.Launch$.launch(Launch.scala:117)
[error] 	at xsbt.boot.Launch$.apply(Launch.scala:18)
[error] 	at xsbt.boot.Boot$.runImpl(Boot.scala:56)
[error] 	at xsbt.boot.Boot$.main(Boot.scala:18)
[error] 	at xsbt.boot.Boot.main(Boot.scala)
[error] Caused by: java.lang.OutOfMemoryError: GC overhead limit exceeded
[error] 	at scala.collection.mutable.ResizableArray.$init$(ResizableArray.scala:32)
[error] 	at scala.collection.mutable.ArrayBuffer.<init>(ArrayBuffer.scala:49)
[error] 	at scala.tools.nsc.classpath.AggregateClassPath.mergeClassesAndSources(AggregateClassPath.scala:97)
[error] 	at scala.tools.nsc.classpath.AggregateClassPath.list(AggregateClassPath.scala:85)
[error] 	at scala.tools.nsc.symtab.SymbolLoaders$PackageLoader.doComplete(SymbolLoaders.scala:271)
[error] 	at scala.tools.nsc.symtab.SymbolLoaders$SymbolLoader.complete(SymbolLoaders.scala:220)
[error] 	at scala.tools.nsc.symtab.SymbolLoaders$SymbolLoader.load(SymbolLoaders.scala:236)
[error] 	at scala.reflect.internal.Symbols$Symbol.$anonfun$typeParams$1(Symbols.scala:1757)
[error] 	at scala.reflect.internal.Symbols$Symbol.completeTypeParams$1(Symbols.scala:1757)
[error] 	at scala.reflect.internal.Symbols$Symbol.typeParams(Symbols.scala:1764)
[error] 	at scala.reflect.internal.Types.isRawIfWithoutArgs(Types.scala:3909)
[error] 	at scala.reflect.internal.Types.isRawIfWithoutArgs$(Types.scala:3909)
[error] 	at scala.reflect.internal.SymbolTable.isRawIfWithoutArgs(SymbolTable.scala:18)
[error] 	at scala.reflect.internal.tpe.TypeMaps$$anon$1.apply(TypeMaps.scala:325)
[error] 	at scala.reflect.internal.tpe.TypeMaps$$anon$1.apply(TypeMaps.scala:322)
[error] 	at scala.reflect.internal.Symbols$Symbol.modifyInfo(Symbols.scala:1563)
[error] 	at scala.reflect.internal.Symbols$Symbol.cookJavaRawInfo(Symbols.scala:1712)
[error] 	at scala.tools.nsc.typechecker.Infer$Inferencer.checkAccessible(Infer.scala:272)
[error] 	at scala.tools.nsc.typechecker.Typers$Typer.makeAccessible(Typers.scala:579)
[error] 	at scala.tools.nsc.typechecker.Typers$Typer.$anonfun$typed1$56(Typers.scala:5007)
[error] 	at scala.tools.nsc.typechecker.Typers$Typer$$Lambda$10739/1676846511.apply(Unknown Source)
[error] 	at scala.tools.nsc.typechecker.Typers$Typer.silent(Typers.scala:700)
[error] 	at scala.tools.nsc.typechecker.Typers$Typer.typedSelect$1(Typers.scala:5007)
[error] 	at scala.tools.nsc.typechecker.Typers$Typer.typedSelectOrSuperCall$1(Typers.scala:5064)
[error] 	at scala.tools.nsc.typechecker.Typers$Typer.typedInAnyMode$1(Typers.scala:5587)
[error] 	at scala.tools.nsc.typechecker.Typers$Typer.typed1(Typers.scala:5603)
[error] 	at scala.tools.nsc.typechecker.Typers$Typer.runTyper$1(Typers.scala:5640)
[error] 	at scala.tools.nsc.typechecker.Typers$Typer.typedInternal(Typers.scala:5672)
[error] 	at scala.tools.nsc.typechecker.Typers$Typer.body$2(Typers.scala:5613)
[error] 	at scala.tools.nsc.typechecker.Typers$Typer.typed(Typers.scala:5618)
[error] 	at scala.tools.nsc.typechecker.Typers$Typer.typedQualifier(Typers.scala:5723)
[error] 	at scala.tools.nsc.typechecker.Typers$Typer.typedQualifier(Typers.scala:5729)
[error] java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError: GC overhead limit exceeded
[error] Use 'last' for the full log.
Exited with code 1

(Boyang) #2

+1 I’m seeing similar errors! Haven’t found a fix

OP: have you tried increase memory limit for sbt?


(Marcos Pereira) #3

Hi @lejoow,

What are the tests doing? What happens if you only update Lagom but not sbt? Splitting the updates in two separated changes can help to better identify where is the problem.

Best.


(Joo) #4

Thanks Marco. We were not running any tests during the deployment but it still happened.