A quick guide to Spring Tests optimization

Podés leer este post en español aquí.

This list of pain points about test performance and the recommendations on how to deal with them came from my experience with slow tests in a Kotlin + Spring Boot + Gradle project with about 1500 tests. We started with 13 minutes per run and ended with 5 minutes (while running them locally).

Why is it important to have fast tests?

Having slow tests can be a headache for example in these cases:

• When you are doing TDD (Test-driven development) and you need to run a test several times (changing something small and running it again and again)
• When you are changing the code and you want to run all tests to see if everything is still working fine, or doing regression testing

Pain Points

• JVM initialization time
• Tests depending on Spring Boot (integration tests among them)
• Some practices that force to re-create the Spring context (annotations like @MockBean, @SpyBean, @DirtiesContext, etc)

Solutions

JVM Initialization Time

You've probably seen this message when you run tests locally:

This is the Java Virtual Machine (JVM) starting. Sometimes it takes too long, and when you want to run only one simple unit test it is a struggle. Doing some research, I noticed that running tests with the JUnit configuration instead of the default configuration (Gradle) improves this initialization time a lot.

To run your tests with JUnit instead of Gradle:

Go to Preferences → Build, Execution, Deployment → Gradle and set the run configuration to "Choose per test" and when you want to run a test, a test class or a suite, you can choose to run them with JUnit:

Spring Context

Some tests need to initialize the Spring context before they run. Those are the ones that have, for example, the annotation @SpringBootTest above.

This can take too long, so we should try to reuse the Spring Boot context between test classes, to start it fewer times.

The context is reused by default unless you do some things like:

• Using Mockito or Mockk annotations like @MockBean or @SpyBean, annotations that generate their own custom context
• Using the @DirtiesContext annotation that explicitly creates a new context (example of common usage: in case you are changing singleton classes in two test classes and you don't want to affect one test class with the other one, you add this annotation).

What’s the solution?

• Try different alternatives to avoid using @MockBean & @SpyBean.
  
  Two examples:

  1) If you have a Controller Test that wants to test the case where a 404 is returned when the Car you are asking for doesn’t exist in the DB:

  Don’t do this ❌

@SpringBootTest
@AutoConfigureMockMvc
class CarControllerTest {
    @Autowired
    private lateinit var mvc: MockMvc
    @MockkBean
    private lateinit var carService: CarService

    @Test
    fun `a 404 code is returned if you are asking for a non-existent car`() {
        every { carService.find(1) } throws NoSuchCarException()

        mvc.perform(MockMvcRequestBuilders.get("/cars/").param("carId", "1"))
        .andExpect(MockMvcResultMatchers.status().`is`(HttpStatus.NOT_FOUND.value()))
    }
}

 Do this ✅

  Don't mock neither the service nor the repository, instead: ask for a Car you know doesn’t exist in the DB (for example id = 9999), or delete the car with id = 1 through the CarRepository before asking for it through the Controller.

  Doing this is an advantage, as you are testing this functionality in a more realistic way and without using mocks

  2) In some cases you can replace a @MockBean by creating a normal mock, passing it through the constructor of the class you will need, instead of using @Autowired for that class. You will also get rid of the @SpringBootTest annotation:

  Don’t do this ❌

class CarServiceTest {
    @MockkBean
    private lateinit var weatherGateway: WeatherGateway
    @Autowired
    private lateinit var carService: CarService

  Do this ✅

class CarServiceTest {
    private var weatherGateway: WeatherGateway = mockk()
    private var carService = CarService(weatherGateway)

• If you have MockBeans that are difficult to get rid of, you can create a parent class with those mocks and inherit from it in all SpringBoot test classes, so that the context is always the same custom context.
  
  Example: you have a class which has a mockbean "A", and another one which has a mockbean "B". In that case there are 2 different custom contexts ❌
  
  ✅ The solution can be creating a parent class that has both mockbean "A" and mockbean "B", so that the custom context is only one in both children classes.
  
  ⚠️ Be aware that some test classes may need the real instance of that class and not the mock, so you should probably unmock it on the class you are mocking it after it’s used

• Avoid the usage of annotations like @DirtiesContext by identifying which beans were shared and changed between test classes and cleaning those beans up before another class uses them.
  Example:
  
  We noticed that we were using the same bean in many test classes: a JPA Repository. And although we were deleting everything in the repo after each test class, we were having some flaky tests. This is because no matter if you delete objects on the repo, the ids keep incrementing, and we were having some tests that had a dependency to a specific id.
  
  For example: I create something on the repo, and then I check something to the id 1 because it's the only entity in the repo… wrong assumption! ❌
  
  ✅ The solution is doing tests that don't depend on a specific and hardcoded id.
  An example of this:

@Test
fun "an exception is thrown if the car doesn't have enough fuel to travel"() {
    val car = carRepository.save(Car(type = CarType.FORD_FOCUS, remainingFuelLiters = 35))
    
    val exception = assertThrows<NotEnoughFuelException> { carService.travel(car.id, 400) }
    assertEquals("Your car has not enough fuel to travel 400 km", exception.message)
}

Other useful things

Custom Run/Debug Configurations & Tagging

You can create your own JUnit configurations, to run only the tests of a specific class, a specific directory, package, and more.

You can also create a tag over a specific test class, by using the JUnit @Tag:

@SpringBootTest
@Tag(“integration-test”)
class CarServiceTest {

You can then create a configuration combining different tags (the tags can be used as boolean fields, so all boolean operators can be used). For example, this configuration ignores the integration tests:

You can also check the "Store as project file" option to save a configuration and add it to the repo so that everyone else has it.

Profiling

Measuring how long it takes to run the tests and what takes longer. You can do that in a few ways. However, the numbers may not be really representative of what’s going on. For example, the IDE sometimes states the tests run in 1’ more or less. But that’s not the case. It’s just that it’s not computing the build time and the application context creation times.

To see reports with more details and real-time numbers you can export them with these 2 IDE options:

Recommendations & Conclusions

• The pain points that were mentioned here are not the only pain points that you can encounter, but they can decrease a lot the performance of your tests
• Try to use JUnit instead of Gradle to run tests, mostly if you are running simple suites and you want to know the result faster
• Initializing the Spring context is not cheap, do it only if it's necessary. The usage of annotations like @SpringBootTest is generally associated with Integration tests, and to avoid falling into the Ice Cream Cone anti-pattern, it is recommended to have more unit tests than integration tests, so don't put @SpringBootTest annotation everywhere
• If you are using @SpringBootTest, try to reuse the context as much as you can, avoiding the anti-patterns I've mentioned before and doing some research if you see that it's not being reused and you are not using the annotations I said (to understand in which tests the Spring Context is being initialized, you should look in which test class the SPRING logo appears)
• Use mocks only if it’s necessary, for example in case the context is difficult to reproduce. Don’t mock everything because you won’t end up testing anything at all.