Be more lenient with test error messages

[clebergnu]

On latest Fedora rawhide, current setuptools usage will produce warning messages on STDERR. This makes the checking more lenient with extra content that may show up.

Summary by CodeRabbit

• Tests
  • Updated error message validation in functional tests to use more flexible pattern matching instead of exact output comparison.

_{✏️ Tip: You can customize this high-level summary in your review settings.}

[coderabbitai]

Walkthrough

This pull request modifies two test functions in selftests/functional/basic.py. The tests test_empty_test_list and test_not_found are updated to change their error verification approach from asserting exact equality of stderr output to asserting that stderr contains specific error message substrings. This relaxes the strictness of error output validation in both test cases.

Estimated code review effort

🎯 2 (Simple) | ⏱️ ~8 minutes

• Homogeneous change pattern applied consistently across two test functions
• Only one file affected with modifications to similar assertion logic
• Straightforward refactoring from equality checks to substring containment checks
• No new logic or complex reasoning required

Pre-merge checks

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%.	You can run @coderabbitai generate docstrings to improve docstring coverage.

✅ Passed checks (2 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title accurately describes the main change: making test error message checking more lenient by converting from exact equality assertions to substring matching in error verification.

---

📜 Recent review details

Configuration used: CodeRabbit UI

Review profile: CHILL

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between b81eeef and 9fea9d2.

📒 Files selected for processing (1)

• selftests/functional/basic.py (2 hunks)

⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (47)

• GitHub Check: rpm-build:fedora-41-aarch64
• GitHub Check: rpm-build:fedora-42-x86_64
• GitHub Check: rpm-build:fedora-41-x86_64
• GitHub Check: rpm-build:fedora-43-x86_64
• GitHub Check: rpm-build:fedora-rawhide-x86_64
• GitHub Check: rpm-build:fedora-41-ppc64le
• GitHub Check: rpm-build:fedora-41-s390x
• GitHub Check: rpm-build:centos-stream-9-x86_64
• GitHub Check: rpm-build:epel-9-x86_64
• GitHub Check: Linux with Python 3.12
• GitHub Check: Linux with Python 3.11
• GitHub Check: Linux with Python 3.13
• GitHub Check: Linux with Python 3.10
• GitHub Check: rpm-build:fedora-42-x86_64
• GitHub Check: rpm-build:fedora-rawhide-x86_64
• GitHub Check: rpm-build:fedora-41-x86_64
• GitHub Check: rpm-build:fedora-41-s390x
• GitHub Check: rpm-build:fedora-41-ppc64le
• GitHub Check: rpm-build:epel-9-x86_64
• GitHub Check: rpm-build:fedora-41-aarch64
• GitHub Check: rpm-build:centos-stream-9-x86_64
• GitHub Check: rpm-build:fedora-43-x86_64
• GitHub Check: rpm-build:fedora-rawhide-x86_64
• GitHub Check: rpm-build:fedora-41-x86_64
• GitHub Check: rpm-build:fedora-43-x86_64
• GitHub Check: rpm-build:fedora-41-ppc64le
• GitHub Check: rpm-build:centos-stream-9-x86_64
• GitHub Check: rpm-build:epel-9-x86_64
• GitHub Check: rpm-build:fedora-42-x86_64
• GitHub Check: rpm-build:fedora-41-s390x
• GitHub Check: rpm-build:fedora-41-aarch64
• GitHub Check: Egg task ubuntu:24.04
• GitHub Check: Egg task debian:11.0
• GitHub Check: Egg task ubuntu:22.04
• GitHub Check: Egg task ubi:8.8
• GitHub Check: Egg task fedora:41
• GitHub Check: Fedora selftests
• GitHub Check: Podman spawner with 3rd party runner plugin
• GitHub Check: Version task ubuntu:22.04
• GitHub Check: Version task debian:12.4
• GitHub Check: Build Package (wheel/tarball) for Python 3.9
• GitHub Check: Version task ubi:9.2
• GitHub Check: Version task ubi:8.8
• GitHub Check: Build Package (wheel/tarball) for Python 3.10
• GitHub Check: Static checks
• GitHub Check: macOS with Python 3.11
• GitHub Check: Code Coverage (3.11)

🔇 Additional comments (2)

selftests/functional/basic.py (2)

644-657: Lenient stderr assertion for empty test list matches intent

Using assertIn(expected_bytes, result.stderr) keeps the test focused on the specific error message while allowing extra content (such as setuptools warnings) on stderr. The bytes literal concatenation and newline handling look correct and consistent with the rest of the suite.

---

659-671: Substring check in test_not_found is correctly relaxed

Switching from exact equality to assertIn(expected_bytes, result.stderr) appropriately makes this test resilient to additional stderr output while still verifying the full, two-line “No tests found…” message. The use of bytes and the expected text remain precise enough for regression detection.

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 73.47%. Comparing base (b81eeef) to head (9fea9d2).
⚠️ Report is 2 commits behind head on master.

Additional details and impacted files

@@           Coverage Diff           @@
##           master    #6252   +/-   ##
=======================================
  Coverage   73.46%   73.47%           
=======================================
  Files         206      206           
  Lines       22497    22497           
=======================================
+ Hits        16527    16529    +2     
+ Misses       5970     5968    -2     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[richtja]

LGTM, thank you