Repository navigation
Deadlock in tests running in CI #284
Description
Hi,
I'm encountering deadlock in CI environment (Docker on Linux, Jenkins job executor runs pipeline). I have 26 integration test suites, each creates docker container with Mongo inside for test cases inside. Suites are executed sequentially. Problem occurs occasionally under 1.1.7 and steadily under 1.1.8, however adding logging appender to org.testcontainers package makes problem go completely away under both versions, which makes this a heisenbug. I'm including container set up code from testing harness:
package com.redacted
import groovy.transform.PackageScope
import org.testcontainers.containers.GenericContainer
@PackageScope
class MongoDBContainer extends GenericContainer {
MongoDBContainer(Integer mongoHostPort) {
super('mongo:3.2')
addFixedExposedPort(mongoHostPort, 27017)
}
}
This class is instantiated in turn in abstract test suite class which is used as base for concrete test suites using those methods (testing framework is Spock, hence Groovy):
def setup() {
dbContainer = new MongoDBContainer(setUpMongoContainer())
//... application setup code ...
dbContainer.start()
}
def cleanup() {
dbContainer.stop()
// ... application teardown code ...
}
private static setUpMongoContainer() {
def mongoHostPort = randomAvailablePort()
System.setProperty("com.redacted.db.connectionstring", "mongodb://127.0.0.1:$mongoHostPort/RedactedService")
return mongoHostPort
}
private static randomAvailablePort() {
try {
ServerSocket socket = new ServerSocket(0)
int httpPort = socket.getLocalPort()
socket.close()
return httpPort
} catch (exception) {
throw new RuntimeException(exception)
}
}
Can you advise and help track down the problem? Right now the only solution that helps is having logging enabled, which is hardly optimal.
Activity
rnorth commentedon Feb 2, 2017
@lbialy hmm, we'll have to try and trace. I'll see if there's any place where logging might be breaking a deadlock - seems like a good clue.
FWIW in your example I think you might be able to simplify things by doing something like:
With this, there would be no need to find a random port yourself - you could let Docker find one for you, and Testcontainers tell you using
getMappedPort. Unless, of course, this has to be done for some other reason?+1 for @asafm here perhaps ;)
lbialy commentedon Feb 2, 2017
No, in this case there wasn't really any other reason than to guarantee different ports,
getMappedPortwould do that job too. Anyway, I'll wait for your findings, currently CI has timeout and kills hanging jobs, but that's just a workaround and jobs should pass deterministically.lbialy commentedon Feb 17, 2017
Hey @rnorth, any update on this? I'm still experiencing deadlocks in CI. 😢
rnorth commentedon Feb 18, 2017
@lbialy sorry for the delay :(. I've tried to reproduce today and am not yet seeing a deadlock. Just to clarify, could you possibly send your
logback-test.xmlfile, as I'm not clear whether you have noorg.testcontainersappender at all (letting it log at the default level) or if you're using one with theOFFlevel...Also, for the purposes of having something to analyze if/when I do see the deadlock, I've added the following (hacky and ugly) code to my test class to dump thread state periodically:
The idea is that, if a deadlock happens, we'll have some stack traces to tell us where we're stuck. If we're struggling to reproduce then adding something like this to your code may at least give us some clues.
Ttest for #284
rnorth commentedon Mar 13, 2017
Hi @lbialy, did the example code or debugging help to track down or resolve the deadlock?
bsideup commentedon Apr 11, 2018
Closed due to inactivity. Anyway, sounds like an issue of docker-java, but @lbialy feel free to reopen if it still happens to you with the latest release ( 1.7.0 )