Skip to content

Commit ad331d2

Browse files
committed
Remove continue-on-error for benchmarks in CI
* Otherwise new failures do not get noticed. * Instead, just exclude the benchmarks known to fail. * Relates to 6d43a4c. There has been a couple failures which were due to the harness assuming Ractor is defined, this CI job helps catch such issues. There is also CRuby-specific code in run_benchmarks.rb, so this tests it's properly guarded.
1 parent cc0335f commit ad331d2

File tree

1 file changed

+13
-3
lines changed

1 file changed

+13
-3
lines changed

.github/workflows/test.yml

Lines changed: 13 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -43,7 +43,18 @@ jobs:
4343
strategy:
4444
fail-fast: false
4545
matrix:
46-
ruby: [ruby, head, truffleruby]
46+
include:
47+
- { ruby: ruby }
48+
- { ruby: head }
49+
- {
50+
ruby: truffleruby,
51+
# knucleotide: needs fork
52+
# lobsters: commonmarker fails to install
53+
# railsbench: NoMethodError: undefined method `[]=' for class Fiber (NoMethodError)
54+
# ruby-lsp: rbs_extension.so: undefined symbol: ruby_vm_at_exit
55+
# shipit: cannot load such file -- pty
56+
extra_args: "--excludes=knucleotide,lobsters,railsbench,ruby-lsp,shipit"
57+
}
4758
if: ${{ github.event_name != 'schedule' || github.repository == 'ruby/ruby-bench' }}
4859
steps:
4960
- uses: actions/checkout@v3
@@ -53,12 +64,11 @@ jobs:
5364
ruby-version: ${{ matrix.ruby }}
5465

5566
- name: Test run_benchmarks.rb
56-
run: ./run_benchmarks.rb
67+
run: ./run_benchmarks.rb ${{ matrix.extra_args }}
5768
env:
5869
WARMUP_ITRS: '1'
5970
MIN_BENCH_ITRS: '1'
6071
MIN_BENCH_TIME: '0'
61-
continue-on-error: ${{ matrix.ruby == 'truffleruby' }}
6272

6373
benchmark-ractor:
6474
runs-on: ubuntu-latest

0 commit comments

Comments
 (0)