Run a websocket server in a multi-jvm test

Akka Akka HTTP
1

Hi all,

I have a multi-jvm integration test that involves 3 JVMs and runs several tests.
Now, these 3 JVMs need to communicate with an “external” server.
Currently, i start this server beforehand, then run the integration test, and after the tests i close the server.

To avoid starting and closing the server manually i want to let the multi-jvm test start by running the server, perform the multi-jvm tests, and then close the server. However, the JVMs cannot connect to the server when it is started from one of the JVMs spawned by the multi-jvm test.

The server is written using akka-http and accepts incoming websocket requests:

object DistributedLockServer {
  implicit val system = ActorSystem("DistributedLocks")
  implicit val materializer = ActorMaterializer()
  implicit val executionContext = system.dispatcher

  val route = path("ws-lock") {
    parameter(Symbol("user")) { user =>
      handleWebSocketMessages(WebSocket.listen(user))
    }
  }

  def main(args: Array[String]): Unit = {
    val bindingFuture = Http().bindAndHandle(route, "0.0.0.0", 8080)

    println(s"Running server on port $port\nPress RETURN to stop...")
    StdIn.readLine()
    bindingFuture
      .flatMap(_.unbind())
      .onComplete(_ => system.terminate())
  }
}

object WebSocket {
  def listen(user: String): Flow[Message, Message, _] = {
    println(s"User connected: $user")
    val inbound: Sink[Message, Any] = Sink.foreach(msg => handleMessage(msg, user))
    val outbound: Source[Message, SourceQueueWithComplete[Message]] = Source.queue[Message](16, OverflowStrategy.fail)

    Flow.fromSinkAndSourceMat(inbound, outbound)((_, outboundMat) => {
      clientConnections += user -> outboundMat.offer
      NotUsed
    })
  }

  private def handleMessage(msg: Message, user: String): Unit = {
    // ...
  }
}

The multi-jvm test runs 3 JVMs and one of them first starts the server.
However, the other JVMs cannot connect to the server:

class DistributedStoreMultiJvmNode1 extends DistributedStoreTest
class DistributedStoreMultiJvmNode2 extends DistributedStoreTest
class DistributedStoreMultiJvmNode3 extends DistributedStoreTest

class DistributedStoreTest extends MultiNodeSpec(ThreeNodeConfig) with ImplicitSender with ScalaTestMultiNodeSpec {
  import ThreeNodeConfig._

  // tell the testkit how many nodes we expect to participate at first
  override def initialParticipants = 3

  "The distributed store" must {
    "start the lock server" in within(5.minutes) {
      runOn(node1) {
        val serverThread = new Thread {
          override def run: Unit = {
            DistributedLockServer.main(Array())
          }
        }

        serverThread.start()
      }
      Thread.sleep(5*1000*60) // 5 minutes
    }
  }
}

object ThreeNodeConfig extends MultiNodeConfig {
  val node1 = role("Alice")
  val node2 = role("Bob")
  val node3 = role("Charlie")

  testTransport(on = true)

  // common configuration for all nodes
  commonConfig(ConfigFactory.parseString(
    """
      |akka.loglevel=INFO
      |akka.actor.provider = cluster
      |akka.actor.allow-java-serialization = on
      |akka.remote.artery.enabled = on
      |akka.coordinated-shutdown.run-by-jvm-shutdown-hook = off
      |akka.coordinated-shutdown.terminate-actor-system = off
      |akka.cluster.run-coordinated-shutdown-when-down = off
    """.stripMargin))
}

In this example i start the server and wait 5 minutes to give me time to manually debug the problem.
I then try to connect to the server using the wsc terminal client for websockets:

$ wsc ws://127.0.0.1:8080/ws-lock
Error: connect ECONNREFUSED 127.0.0.1:8080
    at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1113:14)
Emitted 'error' event at:
    at ClientRequest.onerror (/usr/local/lib/node_modules/wsc/node_modules/ws/lib/WebSocket.js:711:10)
    at ClientRequest.emit (events.js:182:13)
    at Socket.socketErrorListener (_http_client.js:391:9)
    at Socket.emit (events.js:182:13)
    at emitErrorNT (internal/streams/destroy.js:82:8)
    at emitErrorAndCloseNT (internal/streams/destroy.js:50:3)
    at process._tickCallback (internal/process/next_tick.js:63:19)

So, the server refuses the connection, but i don’t know why.
When i run a regular unit test (with ScalaTest) i can connect to the server,
but for some reason the connection is refused in the multi-jvm test.
Is there some configuration i need to do to tell the multi-jvm test to allow connections on port 8080?

2

Hard to say what the problem is there. Sometimes it can be a problem with IPv6 vs. IPv4 vs. localhost. There’s nothing inherent in akka-http that would explain it.