Repository navigation
Allow option to keep the containers alive #109
Description
Activity
You could subclass it and overwrite the
__del__, since that gives you an option to inject the behaviour that you require. That's the approach we took when we needed an alternative behaviour for the Selenium container wrapper.@Can-Sahin, makes sense to want to keep the container alive to speed up tests. Having said that, I'm not sure how you'd be able to connect to the existing container after the reference to the instance variable has been discarded.
If you're using pytest, you can use global fixtures to reuse the same container across all test runs within the same process. Reusing an existing container across different test runs/processes would require persisting information about the container across runs---container management should probably be outside the scope of this project.
Reacted by Vinay Keerthi, Robbie Palmer, dex-kartik and Jozsef LazarClosing this one for now but feel free to reopen if using fixtures with a different scope cannot address the problem.
I figured out the same workaround mentioned in #109 (comment) , then found this issue about keeping containers alive.
I think this is worth documenting at least. I was surprised to see that my containers were getting deleted even when I explicitly didn't start them with a context. I had to dig around the code of testcontainers to figure out that it was
__del__doing this.Also note that the .NET implementation of testcontainers doesn't auto-delete containers if the instance isn't wrapped in the equivalent of a
withstatement. There's no right or wrong here but it would be nice if testcontainers behaved roughly the same across languages. I'm not saying which impl should be changed 🙂I'm not sure how you'd be able to connect to the existing container after the reference to the instance variable has been discarded.
I've used the docker client for Python to get the existing container if available+running then get its port, it's just a few lines of code.
container management should probably be outside the scope of this project.
Isn't this what testcontainers-java are managing with testcontainers/testcontainers-java#781 ?
Reacted by NingúSounds like there is sufficiently broad interest in this feature. We could add a
remove: bool = Truekeyword argument to the constructor and keep the container alive ifnot remove. Note that a PR would now need to modify theatexitregistration once #208 is merged.Reacted by Mauricio Scheffer, Ningú, Mason Flint and Antoine MéausooneI am testing a sort of ETL that connects to quite a lot of testcontainers and this feature would be awesome.
I imagine a workflow similar to the reutilization of DB schemas in pytest-django: an optional flag telling whether to try to reuse already running containers and a second flag telling to stop and respawn them. https://pytest-django.readthedocs.io/en/latest/database.html#example-work-flow-with-reuse-db-and-create-db Sorry if this idea is very tailored for pytest.
Yet, for CI environments it would be ideal if any containers were dropped anyway, because some CI systems like jenkins may use long-lived build agents and littering a bunch of containers in every test run would come unhealthy. Maybe the somehow ubiquitous
CIenvironment variable could be honored as an override?Maybe this should be up to the user, but this library could document a canonical snippet to get it working, as well as redesigning the
__del__method so that it was possible out-of-the-box without major hacks.I had a need for this (well, not a need, just annoyance that running even a single test takes couple of seconds). Since I was not ready to give up testing using real database (in my case Postgres) and wanted to continue using testcontainer, my solution is below. I am sharing it in case someone else needs it, but I am happy to contribute with PR if maintainers think solution is acceptable (which would be more simple to implement directly in the project, since there would be no need to subclass
DockerClientand no need to reimplement same things in subclass if base class can be changed, no accessing to semi-private members and functions, etc).My solution requires that containers have name defined. Than, it uses that name to reuse container across the runs. That way, there is no need for testcontainers to know which container to reuse, it is moved to user.
I am using python 3.12, so some syntax might not work in older versions, that can be easily fixed in a real PR.
from __future__ import annotations import os import logging import atexit from docker.models.containers import Container from testcontainers.postgres import PostgresContainer from testcontainers.core.container import DockerContainer from testcontainers.core.docker_client import DockerClient as OriginalDockerClient, _stop_container from typing import override, Self logger = logging.getLogger(__name__) class DockerClient(OriginalDockerClient): @override def run(self, image: str, command: str = None, environment: dict = None, ports: dict = None, detach: bool = False, stdout: bool = True, stderr: bool = False, remove: bool = False, stop_at_exit: bool = True, **kwargs) -> Container: """ Default implementation with configurable container stopping atexit. """ container = self.client.containers.run( image, command=command, stdout=stdout, stderr=stderr, remove=remove, detach=detach, environment=environment, ports=ports, **kwargs ) if stop_at_exit: atexit.register(_stop_container, container) return container def find_container_by_name(self, name: str) -> Container | None: for cnt in self.client.containers.list(all=True): if cnt.name == name: return cnt return None class PermanentContainer(DockerContainer): """ Docker testing container that has ability to keep using same container for multiple test runs. If it is configured so, it does not destroy container at the end of test run. Same container is reused next time. It is required to configure container name when this feature is used, since name of the container is used to find container from previous run. To enable, pass "keep_container" to init or use "with_keep_container" function. Default value is based on detection of CI environment. CI environment is considered every environment that has "CI" environment variable set. If current env is CI, keep_container is False by default, otherwise it is True by default. """ def __init__(self, *args, **kwargs): self._keep_container = kwargs.pop("keep_container", self._default_keep_container()) super().__init__(*args, **kwargs) @override def start(self) -> Self: # return as fast as possible, if not using keep container if not self._keep_container: self._create_and_start_container() return self if self._keep_container and not self._name: raise Exception("If keep_container is used, name of container must be set") existing_container = self.get_docker_client().find_container_by_name(self._name) if existing_container: if existing_container.status != "running": existing_container.start() logger.info("Using existing container %s", existing_container.id) self._container = existing_container return self # since container is not found, this is probably the first run self._create_and_start_container() return self def _create_and_start_container(self): # copy of super().start() but with some parameter overrides. self._container = self.get_docker_client().run( self.image, command=self._command, detach=True, environment=self.env, ports=self.ports, name=self._name, volumes=self.volumes, stop_at_exit=not self._keep_container, **self._kwargs ) @override def stop(self, force=True, delete_volume=True): if self._keep_container: return return super().stop(force, delete_volume) @override def get_docker_client(self) -> DockerClient: return DockerClient() def with_keep_container(self, keep: bool = True): self._keep_container = keep @staticmethod def _default_keep_container() -> bool: return os.getenv("CI", None) is None class PermanentPostgresContainer(PostgresContainer, PermanentContainer): """ Postgres variant of permanent container. See docs for PermanentContainer for details. """ pass
Reacted by David Ankin and KoundReacted by Ningú, David Ankin and KoundIn case it's a good source of inspiration, Testcontainers for Java implemented reusable containers (description, PR).
My use case is a bit different than the one discussed above. I'd like to run a pytest test suite. When I'm running the tests locally, I'd like to bring up a Postgres container if it's not already running, or reuse it if it is running to speed up my test run. When I'm running the tests on CI, I'd like to use a GitHub Actions Postgres service container, and not try to bring up a Postgres container at all. I figure both use cases can be solved by "don't bring up a container if one exists".
Reacted by Ryan HobanWhen i was introduced to testcontainers at work the first time, we implemented logic to use local db and fall back on testcontainers; the api has changed and
reuseis great - PRs would be welcome to implement it.3 remaining items
@matthiasschaub it sounds like a great plan - happy to review if PR is submitted
Reacted by Matthias Schaub- added a commit that references this issue
on Jul 2, 2024 Hi, would be great to have this feature in a release! Some containers take quite a long time to be initialized and during development this results in quite some time being wasted waiting for those on test re-runs.
Reacted by Matthias Schaub, Dmytro Hrankin, Jake, Jentsch, Johannes Rueschel, Pascal Mathis, Alex Ioannidis and Gleb Ignatev+1 for this feature, if you need any help, please let me know :)
Reacted by Matthias SchaubI could pick up the PR again - respond to comments and bring it up to main. Would that be welcome?
i can technically do that myself - i am getting a pretty big backlog here again and i think i need to take a couple of days over the upcoming thanksgiving holiday and work through it. as long as i dont need to open any pr's myself i think i will be able to get all that done at once (we have a PR approver requirement).
Reacted by Matthias SchaubI brought the PR #636 up to
main.Reacted by CorneeSeanReacted by CorneeSean
It would be nice to keep the docker containers alive for speeding up the test runs. Now, in every test run containers are re-created and can't keep them alive since this block
removes containers when the instance variable is deleted (when the program terminates) and I cannot override it because this also runs without using
with asblock.I don't want my Mysql container to be recreated everytime. I am running tests very frequently and I am waiting couple seconds everytime. Pretty annoying.
I can make a small PR if that makes sense?