Support parallel JUnit 5 tests

https://www.testcontainers.org/test_framework_integration/junit_5/ states:

Note: This extension has only be tested with sequential test execution. Using it with parallel test execution is unsupported and may have unintended side effects.

To me, the entire point of using Test Containers is to speed up parallel builds. I could easily get sequential tests working without it.

Please add support for parallel JUnit builds.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. If you believe this is a mistake, please reply to this comment to keep it open. If there isn't one already, a PR to fix or at least reproduce the problem in a test case will always help us get back on track to tackle this.

This issue is still relevant.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. If you believe this is a mistake, please reply to this comment to keep it open. If there isn't one already, a PR to fix or at least reproduce the problem in a test case will always help us get back on track to tackle this.

This issue is still relevant.

@cowwoc

@cowwoc have you considered using the Singleton Container pattern?

@bsideup

@bsideup This goes against what I am trying to do. I am trying to run multiple tests in parallel, each connecting to a separate docker container. Imagine each docker container running a database instance. I don't want one test influencing the data seen by another test.

I agree we should look into this - the JUnit4 support allowed parallel containers from the beginning, but unfortunately this wasn’t as easy for JUnit5. We should look into this at some point, but to be honest I think our capacity means this will need to come from a community contribution.

_{Sent with GitHawk}

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. If you believe this is a mistake, please reply to this comment to keep it open. If there isn't one already, a PR to fix or at least reproduce the problem in a test case will always help us get back on track to tackle this.

This issue is still relevant

@mfbieber

I still don't understand how the mock server will know what to answer if I just have one container functioning as the mock server.

@mfbieber as the mock server process will be reused between the tests, you would obviously need to ensure that one test does not affect another. This can be done by either clearing MockServer's expectations (see https://mock-server.com/mock_server/clearing_and_resetting.html ) or by using per-test URL prefixes, so that each test will work with a unique set of URLs. This is outside of Testcontainers' responsibilities, as we only provide a container definition for MockServer.

Starting 10s of 100s of containers per test session is possible, but expensive (time and resources wise). MockServer is implemented in Java and (inside the container) runs a Java process that consumes a good amount of memory (400BM or so, last time I checked). Now, if you have 8 CPUs, you will be running 8 tests in parallel, each starting and stopping 400MB process, resulting in 3.2Gb of memory allocated. Some Docker installations may not have that much memory.

tl;dr: Testcontainers does not include any magic and an attempt to start more containers than you have resources on your machine will result in an error (from the containers' side, not in Testcontainers). There is more: starting 4 or 8 (just an example based on a common number of virtual CPU cores available) heavyweight containers at once may result in timeouts because it would consume a lot of CPU.

@bsideup

@bsideup thanks for your reply and explanations. I am aware that it is resource expensive to run that many containers at once.

Since running containers in parallel is possible (and we could close this issue), how about updating the documentation and providing configuration options for doing that?

The static or singleton approach really didn't result in a significant speedup. Having to provide per-test URL prefixes, or ensure otherwise that the mock responses don't affect each other, is also very laborious.

Regarding the connection timeout, could a .waitingFor(Wait.forHealthcheck()) help to wait for the container in the option with the 10s of containers running in parallel?

@bsideup

@bsideup

Starting 10s of 100s of containers per test session is possible, but expensive (time and resources wise).

That's the entire point of this ticket. I don't want a mock server. I want the real thing run in parallel.

@mfbieber Instead of a static instance, maybe it's worth trying a ThreadLocal?

@mfbieber

@mfbieber Instead of a static instance, maybe it's worth trying a ThreadLocal?

Maybe, but as you also said, I also want it to run in parallel. The speedup is not really noticeable with just one mock server.

@mfbieber

@mfbieber Instead of a static instance, maybe it's worth trying a ThreadLocal?

Junit creates a new instance of your test class for each class, so ThreadLocal should not be required.

@cowwoc

Environment:

Docker server version: 20.10.5
Docker API version: 1.41
TestContainers Selenium: v1.15.3
Spring Boot: v2.4.5
JUnit: v5.7.1
OS: MacOS v10.15.7

Issue:

I would also like to see the support for running multiple test containers in parallel, not a singleton container shared between multiple tests running in parallel. I tried @cowwoc's suggestion of ThreadLocal for my Selenium TestContainer, but starting multiple containers in parallel, results in issue with duplicate mount points -

com.github.dockerjava.api.exception.BadRequestException: Status 400: {"message":"Duplicate mount point: /dev/shm"}

	at org.testcontainers.shaded.com.github.dockerjava.core.DefaultInvocationBuilder.execute(DefaultInvocationBuilder.java:237)
	at org.testcontainers.shaded.com.github.dockerjava.core.DefaultInvocationBuilder.post(DefaultInvocationBuilder.java:124)
	at org.testcontainers.shaded.com.github.dockerjava.core.exec.CreateContainerCmdExec.execute(CreateContainerCmdExec.java:33)
	at org.testcontainers.shaded.com.github.dockerjava.core.exec.CreateContainerCmdExec.execute(CreateContainerCmdExec.java:13)
	at org.testcontainers.shaded.com.github.dockerjava.core.exec.AbstrSyncDockerCmdExec.exec(AbstrSyncDockerCmdExec.java:21)
	at org.testcontainers.shaded.com.github.dockerjava.core.command.AbstrDockerCmd.exec(AbstrDockerCmd.java:35)
	at org.testcontainers.shaded.com.github.dockerjava.core.command.CreateContainerCmdImpl.exec(CreateContainerCmdImpl.java:595)
	at org.testcontainers.containers.GenericContainer.tryStart(GenericContainer.java:407)
	at org.testcontainers.containers.GenericContainer.lambda$doStart$0(GenericContainer.java:325)
	at org.rnorth.ducttape.unreliables.Unreliables.retryUntilSuccess(Unreliables.java:81)
	at org.testcontainers.containers.GenericContainer.doStart(GenericContainer.java:323)
	at org.testcontainers.containers.GenericContainer.start(GenericContainer.java:311)
	at com.axiom.turbo.TestContextManager.beforeEach(TestContextManager.java:43)
	at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.lambda$invokeBeforeEachCallbacks$1(TestMethodTestDescriptor.java:159)
	at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.lambda$invokeBeforeMethodsOrCallbacksUntilExceptionOccurs$5(TestMethodTestDescriptor.java:195)
	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
	at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.invokeBeforeMethodsOrCallbacksUntilExceptionOccurs(TestMethodTestDescriptor.java:195)
	at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.invokeBeforeEachCallbacks(TestMethodTestDescriptor.java:158)
	at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:125)
	at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:65)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$5(NodeTestTask.java:139)
	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$7(NodeTestTask.java:129)
	at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:127)
	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:126)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:84)
	at org.junit.platform.engine.support.hierarchical.ForkJoinPoolHierarchicalTestExecutorService$ExclusiveTask.compute(ForkJoinPoolHierarchicalTestExecutorService.java:185)
	at java.base/java.util.concurrent.RecursiveAction.exec(RecursiveAction.java:189)
	at java.base/java.util.concurrent.ForkJoinTask.doExec$$$capture(ForkJoinTask.java:290)
	at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java)

My JUnit platform properties:

junit.jupiter.execution.parallel.enabled=true
junit.jupiter.execution.parallel.mode.default=concurrent

Not sure if the above exception is a limitation with TestContainers or Docker itself. I understand that running multiple containers in parallel can be resource intensive, but it would be nice if it was left to the discretion of the consumer of this library and not enforced, perhaps by allowing setting a fixed number of containers that can be started in parallel.

As an aside, if running multiple TestContainers isn't possible for now, in the case of Selenium is it possible to set multiple browser instances during setup? For example if set to 5, a single container could run 5 tests on 5 Chrome browser instances. That would be another way to achieve parallelism albeit not an ideal one.

Hi @rnorth and @bsideup can you please share what's stopping #3220 from being merged? I see all checks have passed. Is the implementation incomplete?

@testphreak

@testphreak Testcontainers already supports running containers in parallel, so I guess there is a naming confusion with JUnit's parallel mode.

The issue you're getting looks indeed like an issue with Docker, but consider reporting it separately, for BrowserDriverContainer.

Nothing is stopping #3220, we just need time to properly review & merge it :) Thanks for reminding about it 👍

@bsideup

Hi @bsideup, I think I missed it in the documentation.

Testcontainers already supports running containers in parallel

How do you run containers in parallel? Is there an example available. I tried ThreadLocal like I mentioned earlier, but ran into errors.

@bsideup

Hi @bsideup, I think I missed it in the documentation.

Testcontainers already supports running containers in parallel

How do you run containers in parallel? Is there an example available. I tried ThreadLocal like I mentioned earlier, but ran into errors.

@testphreak I pushed an example which I use for my parallel test cases with Testcontainers. Test pollution is a major challenge when running test suits in parallel. I think this solution is a good compromise between running everything in parallel and just running classes in parallel. Sometimes you require a fresh container context on class level and sometimes you require a fresh container context on method level. Maybe this helps for your use-case as well?

@StefanHufschmidt

@StefanHufschmidt Thank you for this. Out of curiosity, what is the overhead of each approach? For example, is FreshContainersPerTestMethod N times slower than FreshContainersPerTestClass where N denotes the number of tests in the class? If the overhead is low enough, then obviously we should all prefer the former.

Any ETA on this one?

@test

Would it be possible to make testcontainers create one database (inside the running db instance, not a new full container) per @test method. That would make the progress lighter (not running several db docker instances) but still allow isolation between tests

Let's say my tests need to start 2 containers (a database + a web server), I wan't to split all my tests in 3 parallel executions. Can't just ask junit5 to split the tests into 3 parallel run, each test running its own container stack - so that each each parallel execution is isolated from the others? (so at the end I'd have 6 containers)

Hi, I'd like to investigate and work on this issue if parallel execution support is still desired.

I'll first verify the current behavior of the JUnit Jupiter extension against the latest default branch, covering parallel test classes and methods as well as static and instance @Container fields. I'll then create a minimal regression test to identify any lifecycle or concurrency problems before proposing a fix.

Since this issue has been open for some time, could a maintainer confirm whether full parallel execution support is still the intended scope, or whether an incremental contribution covering a specific container lifecycle scenario would be preferred?

Based on contribution activity and what I see in TC Slack, it seems fair to say that maintenance of the project has largely stalled.

Regarding this particular issue, most of the failure scenarios could be addressed simply by making start and stop methods synchronized. However, #11702 has remained unreviewed for more than four months.

Navigation Menu

Repository navigation

Support parallel JUnit 5 tests #1495

24 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Support parallel JUnit 5 tests #1495

Description

Activity

stale commented on Aug 20, 2019

cowwoc commented on Aug 21, 2019

stale commented on Nov 19, 2019

cowwoc commented on Nov 19, 2019

bsideup commented on Nov 28, 2019

cowwoc commented on Nov 28, 2019

rnorth commented on Nov 29, 2019

stale commented on Feb 27, 2020

glockbender commented on Mar 10, 2020

24 remaining items

bsideup commented on Feb 1, 2021

mfbieber commented on Feb 1, 2021

cowwoc commented on Feb 1, 2021

mfbieber commented on Feb 1, 2021

big-andy-coates commented on Feb 12, 2021

testphreak commented on Apr 22, 2021

bsideup commented on Apr 22, 2021

testphreak commented on Apr 22, 2021

StefanHufschmidt commented on Jul 21, 2022

cowwoc commented on Jul 21, 2022

driverpt commented on Feb 3, 2023

fabiorosa-sn commented on Apr 28, 2023

laurentthiebaudezm commented on Mar 7, 2024

exijn commented on Aug 20, 2026

remal commented on Aug 21, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions