Java Play2.6
Hello,
I write functional tests for my microservice to test all REST APIs and their input/output. All functional test classes extend ‘BaseTest’, which starts application, embedded Postgres DB, Kafka, … etc. This started out very well. But as the service is growing, I have now 500++ tests, which take 4++ hours to run everytime I want to run regression tests.
I am trying to change the behviour to have one application startup and be shared by all tests (I will manage the data cleanup between tests). Until I did not have much success. Here is my BaseTest:
package base;
import com.google.common.collect.ImmutableMap;
import com.opentable.db.postgres.embedded.EmbeddedPostgres;
import com.opentable.db.postgres.junit.EmbeddedPostgresRules;
import com.opentable.db.postgres.junit.SingleInstancePostgresRule;
import com.salesforce.kafka.test.junit4.SharedKafkaTestResource;
import org.jdbcdslog.ConnectionLoggingProxy;
import org.junit.After;
import org.junit.Before;
import org.junit.ClassRule;
import org.junit.Rule;
import play.Application;
import play.inject.guice.GuiceApplicationBuilder;
import play.test.WithApplication;
import java.sql.Connection;
import java.sql.SQLException;
import java.util.ArrayList;
import java.util.List;
public class BaseTest extends WithApplication {
protected EmbeddedPostgres embeddedPostgres;
private Connection connection;
/**
* We have a single embedded kafka server that gets started when this test class is initialized.
*
* It's automatically started before any methods are run via the @ClassRule annotation.
* It's automatically stopped after all of the tests are completed via the @ClassRule annotation.
*
* This example we start a cluster with 2 brokers (defaults to a single broker) and configure the brokers to
* disable topic auto-creation.
*/
@ClassRule
public static final SharedKafkaTestResource sharedKafkaTestResource = new SharedKafkaTestResource()
.withBrokers(2)
.withBrokerProperty("auto.create.topics.enable", "true");
@Rule
public SingleInstancePostgresRule pg = EmbeddedPostgresRules.singleInstance();
protected Application provideApplication() {
List disabledModules = new ArrayList<>();
disabledModules.add("startup.KafkaConsumerModule");
embeddedPostgres = pg.getEmbeddedPostgres();
return new GuiceApplicationBuilder()
.configure("db.default", ImmutableMap.of(
"url", embeddedPostgres.getJdbcUrl("postgres", "postgres"),
"port", embeddedPostgres.getPort()
))
.configure("play.modules.disabled", disabledModules)
.build();
}
}
Well, 4 hours is a lot! Where is the bottleneck? Is there some component that takes too long to start? Anyway, one thing you can do is to share the application for the test class. WithApplication starts and stops an application for every test, and you may want to start it once if there are your tests don’t require a clean slate every time. So, instead of using WithApplication you can have something like:
import org.junit.AfterClass;
import org.junit.BeforeClass;
import org.junit.Test;
import play.Application;
import play.inject.guice.GuiceApplicationBuilder;
import play.test.Helpers;
class MyTest {
private static Application application;
@BeforeClass
static void startApplication() {
// Use the builder to customize the app creation
// as needed for this test
application = new GuiceApplicationBuilder().build();
Helpers.start(application);
}
@Test
public void test1() {
// test using the application
}
@Test
public void test2() {
// test something else using the same application
}
@AfterClass
static void stopApplication() {
Helpers.stop(application);
application = null;
}
}
The same can be done if you are using WithServer or WithBrowser. You can also disable heavy modules from tests where they aren’t required. For example, based on the code above:
@BeforeClass
static void startApplication() {
// Use the builder to customize the app creation
// as needed for this test
application = new GuiceApplicationBuilder()
.disable(ModuleThatHasBindsForHeavyComponents.class)
.build();
}
Or maybe even mock these heavy components. But, as I said above, better if you can first identify the bottlenecks and start there.
I was faced with the same problem where I tried to have every test self-contained in rems of all its dependencies. e.g : DB, message broker, cache, and of course Play server in fake mode etc.
It worked except for dependency on cassandra which required creating keyspaces and populating the schema and records for every test which took the overall run to 6+ hours (I have 2400+ testcases)
Obviously as a compromise I have to use shared cassandra keyspace.
One more thing is you could try the in-memory H2 DB for unit testcases if you are not much concerned about the actual DB used. Postgres can always be used for integration testing.
Thanks for the info. There are few components initialized for each tests. Application, Kafka, PostgresDB. All these take time.
I am wondering, is there a way to share this application across all test classes? In your example, this application is initialized once per class. I will need to repeat this code in every class (they are many), which I prefer to avoid.
My current code (the one that takes long time), has base class, which is extended by all test class. This is the full base class:
package base;
import com.google.common.collect.ImmutableMap;
import com.opentable.db.postgres.embedded.EmbeddedPostgres;
import com.opentable.db.postgres.junit.EmbeddedPostgresRules;
import com.opentable.db.postgres.junit.SingleInstancePostgresRule;
import com.salesforce.kafka.test.junit4.SharedKafkaTestResource;
import org.jdbcdslog.ConnectionLoggingProxy;
import org.junit.After;
import org.junit.Before;
import org.junit.ClassRule;
import org.junit.Rule;
import play.Application;
import play.inject.guice.GuiceApplicationBuilder;
import play.test.WithApplication;
import java.sql.Connection;
import java.sql.SQLException;
import java.util.ArrayList;
import java.util.List;
public class BaseTest extends WithApplication {
protected EmbeddedPostgres embeddedPostgres;
private Connection connection;
/**
* We have a single embedded kafka server that gets started when this test class is initialized.
*
* It's automatically started before any methods are run via the @ClassRule annotation.
* It's automatically stopped after all of the tests are completed via the @ClassRule annotation.
*
* This example we start a cluster with 2 brokers (defaults to a single broker) and configure the brokers to
* disable topic auto-creation.
*/
@ClassRule
public static final SharedKafkaTestResource sharedKafkaTestResource = new SharedKafkaTestResource()
// Start a cluster with 2 brokers.
.withBrokers(2)
// Eisable topic auto-creation.
.withBrokerProperty("auto.create.topics.enable", "true");
@Rule
public SingleInstancePostgresRule pg = EmbeddedPostgresRules.singleInstance();
protected Application provideApplication() {
//controllers.events.engine.Events.bootstrapServers = sharedKafkaTestResource.getKafkaConnectString();
List disabledModules = new ArrayList<>();
disabledModules.add("startup.KafkaConsumerModule");
System.out.println("calling this");
embeddedPostgres = pg.getEmbeddedPostgres();
System.out.println(embeddedPostgres.getJdbcUrl("postgres", "postgres"));
return new GuiceApplicationBuilder()
//.bindings(new MyModule()) --if you have modules in your service, you need to bind them here
.configure("db.default", ImmutableMap.of(
"url", embeddedPostgres.getJdbcUrl("postgres", "postgres"),
"port", embeddedPostgres.getPort()
))
/*Mocking consumer did not work. I had to disable it, and run my own*/
.configure("play.modules.disabled", disabledModules)
.build();
}
@Before
public void setUp() throws Exception {
}
@After
public void tearDown() throws Exception {
System.out.println("tearDown");
//keepConsumerRunning = false;
//embeddedPostgres.close();
}
/**
* Wrap statements with logger.
* Without calling this, only hibernate statements will be logged, and vanilla java statements will not be logged
* Vanilla java statements are using in unit tests often
* @Link: https://code.google.com/archive/p/jdbcdslog/wikis/UserGuide.wiki#Setup_logging_engine
* @return
* @throws SQLException
*/
protected Connection getConnection() throws SQLException {
connection = embeddedPostgres.getPostgresDatabase().getConnection();
return ConnectionLoggingProxy.wrap(connection);
}
}
Can I reuse your way in my base class and extend it? How can I do that?
Well, I think this is more a JUnit question than a Play question, maybe you need a custom test runner? But the way you described the problem made me think that you need to reconsider your testing approach in the context of having a test pyramid. Having a single class impacting all the tests smells bad in my opinion.
Why do all the tests require all this infra-structure? Can’t the components be tested in isolation instead of requiring everything all the time? Can you refactor some of the integration tests to be unit tests? What if you want/need to run the tests in parallel?
Thanks for the advice. Most of these tests are functional tests testing the REST APIs as black box. I have 50++ REST endpoints. Each one has different scenarios to be tested and most of the time these scenarios require the components like Kafka, DB, … etc. My understanding from Play documentation, extending WithApplication is the recommended way to run functional tests. Or is there another approach you recommend?
Here is one of these tests:
@Test
public void getFreeTextAnswerNonExistingQuestion_NotFound() throws Exception {
Result result = sendGETRequest(NON_EXISTING_QUESTION_ID);
/*Verify response body*/
assertEquals("No Answers found for question with Id: ("+NON_EXISTING_QUESTION_ID+").", ResultParser.getResultBodyAsString(result));
assertEquals(NOT_FOUND, result.status());
}