Skip to content

feat: add support for --export-skipped-tests-file#302

Closed
andheiberg wants to merge 1 commit intomainfrom
add-skipped-reports
Closed

feat: add support for --export-skipped-tests-file#302
andheiberg wants to merge 1 commit intomainfrom
add-skipped-reports

Conversation

@andheiberg
Copy link

@andheiberg andheiberg commented Jan 16, 2025

I've observed failing tests being skipped in a codebase even though the test file wasn't employing the clear and self documenting "skip" method to achieve this. At scale this makes it hard to reason about which tests are actually run and which are skipped.

I wish to create a central file test_todo.yml containing all tests that are allowed to be skipped like so:

skipped:
  - test/unit/a_test.rb

Then I can write a CI job that complains if tests not explicitly marked as skipped centrally is skipped.

SFR has ended up with tests being skipped that shouldn't be skipped. I want to add a CI check confirming that no test wasn't skipped that wasn't explicitly in a test_todo.yml file. To acheive this I need the ci-queue to report which tests were skipped.
@andheiberg andheiberg self-assigned this Jan 16, 2025
@ChrisBr
Copy link
Contributor

ChrisBr commented Jan 16, 2025

I would like to avoid storing more data into Redis and adding more options / complexity unless there is a very good reason.

For skipped tests I'm not sure we need to store it in a distributed way. Maybe we could just do it in a reporter.

class SkipAllowListReporter < Minitest::Reporters::BaseReporter
  def initialize(options = {})
    super
    @allow_list = Set.new(File.read_lines(options.delete(:allow_list)))
  end


  def record(test)
    super
    raise "boom" if test.skipped? && !allow_list.include?(test.name)
  end
end

@ChrisBr ChrisBr closed this Mar 12, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants