Functional tests take very long time. How to optimize that

(Demo) #1

Java Play2.6
Hello,
I write functional tests for my microservice to test all REST APIs and their input/output. All functional test classes extend ‘BaseTest’, which starts application, embedded Postgres DB, Kafka, … etc. This started out very well. But as the service is growing, I have now 500++ tests, which take 4++ hours to run everytime I want to run regression tests.

I am trying to change the behviour to have one application startup and be shared by all tests (I will manage the data cleanup between tests). Until I did not have much success. Here is my BaseTest:

package base;

import com.google.common.collect.ImmutableMap;
import com.opentable.db.postgres.embedded.EmbeddedPostgres;
import com.opentable.db.postgres.junit.EmbeddedPostgresRules;
import com.opentable.db.postgres.junit.SingleInstancePostgresRule;
import com.salesforce.kafka.test.junit4.SharedKafkaTestResource;
import org.jdbcdslog.ConnectionLoggingProxy;
import org.junit.After;
import org.junit.Before;
import org.junit.ClassRule;
import org.junit.Rule;
import play.Application;
import play.inject.guice.GuiceApplicationBuilder;
import play.test.WithApplication;
import java.sql.Connection;
import java.sql.SQLException;
import java.util.ArrayList;
import java.util.List;


public class BaseTest extends WithApplication {
    protected EmbeddedPostgres embeddedPostgres;
    private Connection connection;

    /**
     * We have a single embedded kafka server that gets started when this test class is initialized.
     *
     * It's automatically started before any methods are run via the @ClassRule annotation.
     * It's automatically stopped after all of the tests are completed via the @ClassRule annotation.
     *
     * This example we start a cluster with 2 brokers (defaults to a single broker) and configure the brokers to
     * disable topic auto-creation.
     */
    @ClassRule
    public static final SharedKafkaTestResource sharedKafkaTestResource = new SharedKafkaTestResource()
            .withBrokers(2)
            .withBrokerProperty("auto.create.topics.enable", "true");

    @Rule
    public SingleInstancePostgresRule pg = EmbeddedPostgresRules.singleInstance();

    protected Application provideApplication() {
        List disabledModules = new ArrayList<>();
        disabledModules.add("startup.KafkaConsumerModule");
            embeddedPostgres = pg.getEmbeddedPostgres();
            return new GuiceApplicationBuilder()
                    .configure("db.default", ImmutableMap.of(
                            "url", embeddedPostgres.getJdbcUrl("postgres", "postgres"),
                            "port", embeddedPostgres.getPort()
                    ))
                    .configure("play.modules.disabled", disabledModules)
                    .build();
    }

}

(Marcos Pereira) #2

Hey @Hawk707,

Well, 4 hours is a lot! Where is the bottleneck? Is there some component that takes too long to start? Anyway, one thing you can do is to share the application for the test class. WithApplication starts and stops an application for every test, and you may want to start it once if there are your tests don’t require a clean slate every time. So, instead of using WithApplication you can have something like:

import org.junit.AfterClass;
import org.junit.BeforeClass;
import org.junit.Test;
import play.Application;
import play.inject.guice.GuiceApplicationBuilder;
import play.test.Helpers;

class MyTest {
    private static Application application;
    
    @BeforeClass
    static void startApplication() {
        // Use the builder to customize the app creation 
        // as needed for this test
        application = new GuiceApplicationBuilder().build();
        Helpers.start(application);
    }
    
    @Test
    public void test1() {
        // test using the application
    }

    @Test
    public void test2() {
        // test something else using the same application
    }
    
    @AfterClass
    static void stopApplication() {
        Helpers.stop(application);
        application = null;
    }
}

The same can be done if you are using WithServer or WithBrowser. You can also disable heavy modules from tests where they aren’t required. For example, based on the code above:

@BeforeClass
static void startApplication() {
    // Use the builder to customize the app creation
    // as needed for this test
    application = new GuiceApplicationBuilder()
            .disable(ModuleThatHasBindsForHeavyComponents.class)
            .build();
}

Or maybe even mock these heavy components. But, as I said above, better if you can first identify the bottlenecks and start there.

Best.

1 Like
(Aditya Athalye) #3

Hi @Hawk707

I was faced with the same problem where I tried to have every test self-contained in rems of all its dependencies. e.g : DB, message broker, cache, and of course Play server in fake mode etc.
It worked except for dependency on cassandra which required creating keyspaces and populating the schema and records for every test which took the overall run to 6+ hours (I have 2400+ testcases)

Obviously as a compromise I have to use shared cassandra keyspace.

One more thing is you could try the in-memory H2 DB for unit testcases if you are not much concerned about the actual DB used. Postgres can always be used for integration testing.

HTH
Aditya