RFC: 4.x releases

[totallyzen]

Context

Now that we've got a release pipeline back in action, there is lots we can do to make things nicer.

I am raising this issue in order for community to gather and talk about what should be fixed as priorities and what improvements we want to add.

Known Issues

We know there are several areas we want to improve:

• We want to thin out dependencies where possible - prefer giving a ready container but not be biased on how you should connect to the container. Example: postgres containers enforcing SqlAclhemy, as seen in fix(postgres): Remove SqlAlchemy dependency from postgres container #445
• We want to improve package typing so people have better developer experiences, as seen in Mark packages as typed #305 and the PRs it spawned so far
• Adding new containers, as there is demand for it feat(nats): Add new module #439, fix(kafka): Add redpanda testcontainer module #441, feat(new): add a new Docker Registry test container #389, etc

Note

The list above is not conclusive, please comment on this issue and show your support!

[max-pfeiffer]

@totallyzen Hello Balint, I just created a couple of new issues:

• Bug: Arangodb tests fail on machines with ARM CPU (Apple M1, M2) #449
• Bug: Kafka tests throw errors and run forever on machines with ARM CPU (Apple M1, M2) #450
• Bug: Keycloak tests fail on machines with ARM CPU (Apple M1, M2) #451
• Bug: Elasticsearch tests fail on machine with ARM CPU (Apple M1, M2) #452
• Bug: Mssql tests fail on machines with ARM CPU (Apple M1, M2) #453

The tests for these modules fail on my M2 MacBook. And most likely will fail for everyone else using these testcontainer modules on their machine with ARM CPU.

We already have an older pull request addressing that issue, but that one seems to be outdated and partially obsolet as a couple of issues already became fixed: #357

I would offer to work these issues as it is important to me to have working tests. Also anyone else using these testcontainer modules on their ARM machines would profit from it.

[max-pfeiffer]

@totallyzen PS: I just checked for an M1/M2 GitHub runner. It is actually available since January 2024.

But it comes without a Docker installation and has other quirks. I just did a proof of concept with it: because of the mentioned issues I was not able to run the tests in the pipeline.

[bearrito]

Might be more appropriate to get that in a 5.x.

AFAIK ARM runners aren't generally available yet - https://resources.github.com/devops/accelerate-your-cicd-with-arm-and-gpu-runners-in-github-actions/ . Not sure you'd want to correlate ARM with MAC OS.

In general not too difficult to get that working with github actions using a matrix - https://docs.github.com/en/actions/using-jobs/using-a-matrix-for-your-jobs#using-a-matrix-strategy

[bearrito]

1. Revise documentation after the poetry workflow change. Most of the docs refer to the previous workflow.

2. Correct usage/mental model of extras and community modules. I get the desire for keeping dependencies low. I think the downside of not including a client dependency is while you can verify that the container is running, you can't verify that the connection details you might return are valid for the eventual client. Some want a full uri, others perhaps host and port. Or maybe I didn't expose the correct port etc.

I'm the committer on #439 There is AFAIK on one supported python client to interact with NATS, so it seems natural to include it. I could take or leave any approach.

[max-pfeiffer]

Might be more appropriate to get that in a 5.x.

AFAIK ARM runners aren't generally available yet - https://resources.github.com/devops/accelerate-your-cicd-with-arm-and-gpu-runners-in-github-actions/ . Not sure you'd want to correlate ARM with MAC OS.

In general not too difficult to get that working with github actions using a matrix - https://docs.github.com/en/actions/using-jobs/using-a-matrix-for-your-jobs#using-a-matrix-strategy

Yes, I agree here. I was not aware of the general ARM runner roadmap of GitHub.

And yes, it's quite straight forward to achieve this with a matrix. I already did a little POC: max-pfeiffer#3

[totallyzen]

@max-pfeiffer thanks for the summary on mac! It helps a great deal indeed to know!

For the matrices, I think you'll find it that I've used them extensively in this project already.
We can easily add a PoC PR that piles up all the fixes for MacOS, Windows and ARM and I'm glad there are people who care about this! :)

For me I'm running on a vanilla Ubuntu and not an ARM machine so as you might imagine, testing can be kinda difficult. I also have a young child so time is limited sometimes. If you find me responding somewhat slowly, this is why (managing energy and time).

---

@bearrito Good points overall, I started on the documentation for CONTRIBUTING.md and update the issue templates so we can communicate how things fit together. I'll add all the current active members to that PR as a reviewer so you can have your say! 🙂

As for "what about auto-including clients" I think I am gravitating towards "let's add them as dev dependencies to prove the clients are working, but don't enforce it in live dependencies". This way we can show the users how we intended to use it, without enforcing the specific library (even if there is only one right now).
Deal?

---

I'll check with @kiview on giving you both some level of permissions so you can auto-run some tests yourself on PRs. Not write permissions, just so that your PRs get auto-approval for the GH runs. Same goes for @jankatins as he is also actively contributing with other things!

I think with your dedication and activity you have earned it.

[bearrito]

@totallyzen Deal. I implemented what I think should be satisfactory in a new pr at https://github.com/testcontainers/testcontainers-python/pull/462/files. That could be an overall good approach to managing clients.

[alexanderankin]

found another good fix to get merged #457 for 4.0.x

[alexanderankin]

alright im unsubscribing from notifications from this repo for half a week - its too much - so many people are so busy pushing commits and updating the project with feedback. good job everyone 👍

[alexanderankin]

what if we only maintained semver for core, and we treated all community changes as "fix"?

[kiview]

Maintaining a module ecosystem in itself complex and we have explored different options for this in the past (monorepo vs multirepo, synced versioning, etc.). Since tc-python currently follows the monorepo approach for its modules, they are released as part of a synchronized release job, together with a synchronized version (even if certain modules won't contain changes themself).

For other languages we learned, that this is generally an easy process, not without its drawbacks, but straight forward to maintain.

However in this case, I agree with @alexanderankin that changes in the modules should not be able to significantly contribute to the semver version of core, since I would assume this to be counterintuitive for the majority of users.

[alexanderankin]

@totallyzen @santi any objection to this idea which i have proposed above:

what if we only maintained semver for core, and we treated all community changes as "fix"?

[santi]

@totallyzen @santi any objection to this idea which i have proposed above:

what if we only maintained semver for core, and we treated all community changes as "fix"?

@alexanderankin Having looked through how things are done in tc-java, it seems like they are following the approach with updating (including breaking changes) in community modules as fix/patch changes. At least that's what I gather from this example, where they make breaking changes to the mysql module, which was later released as part of the 1.19.4 patch.

So yes, you have my support. If we were to follow semver for all community module changes we would release a lot of major versions rapidly, which would be like shouting "wolf, wolf" repeatedly and users would stop thinking of that as "dangerous territory" / likely requiring a small code fix.

[totallyzen]

👋 Gotcha! Agreed then, I'll document it in #460 to let users know that we guarantee best-effort semver for community modules.

EDIT: aaand done! 8bfc758

[alexanderankin]

just thinking aloud:

next(?) round of containers:

• feat: Adding support for Cassandra and Scylla #167
• add basic azure cosmos db emulator container #247
• Add Cockroach DB Container Support #281
• fix: Add memcached container #322
• Port BigQueryContainer from testcontainers-java #393

still figuring out environment fixes (dind (#283 still not merged, still some user pains from #388), macos ipv6 localhost nonsense, windows, i started doing some fixes in #457 and #458 but we have had 0 related user reports, which probably means this library doesn't see much use on windows)

[alexanderankin]

cosmos, went out in 4.7.0, cockroachdb also released. for 4.8.0:

fixed dind tests in #622 - going to hopefully also fix dind there and then scylla, BQ, and the others from my last comment - #446 (comment)