chore: Add benchmarking dashboard #7068

TomAFrench · 2025-01-14T23:27:33Z

Description

Problem*

Resolves

Summary*

This PR adds a new benchmarking dashboard at https://noir-lang.github.io/noir/dev/bench/. Results will be added to this on each push to master.

We'll still be benchmarking on each PR however comments will only be posted if there's a regression above a certain percentage (currently set to a quite aggressive 10% but we can dial that back if necessary)

I'll turn off the comment systems for metrics other than gate counts in a separate PR.

Additional Context

Documentation*

Check one:

No documentation needed.
Documentation included in this PR.
[For Experimental Features] Documentation to be submitted in a separate PR.

PR Checklist*

I have tested the changes locally.
I have formatted the changes with Prettier and/or cargo fmt on default settings.

github-actions · 2025-01-15T16:48:18Z

Compilation Report

Program	Compilation Time	%
sha256_regression	1.160s	8%
regression_4709	0.837s	-4%
ram_blowup_regression	16.600s	1%
rollup-root	3.922s	8%
rollup-merge	2.164s	8%
rollup-block-root-single-tx	143.000s	0%
rollup-block-root-empty	2.016s	-6%
rollup-block-root	137.000s	-2%
rollup-block-merge	3.488s	-5%
rollup-base-public	27.840s	-1%
rollup-base-private	10.100s	-3%
private-kernel-tail	1.051s	1%
private-kernel-reset	6.126s	2%
private-kernel-inner	2.174s	2%

github-actions · 2025-01-15T16:48:18Z

Execution Report

Program	Execution Time	%
sha256_regression	0.051s	-2%
regression_4709	0.001s	0%
ram_blowup_regression	0.611s	1%
rollup-root	0.105s	0%
rollup-merge	0.006s	0%
rollup-block-root	37.000s	1%
rollup-block-merge	0.104s	0%
rollup-base-public	1.230s	-1%
rollup-base-private	0.449s	-1%
private-kernel-tail	0.019s	-6%
private-kernel-reset	0.314s	-2%
private-kernel-inner	0.068s	-6%

github-actions · 2025-01-15T18:27:09Z

Execution Memory Report

Program	Peak Memory	%
keccak256	74.630	100%
workspace	123.640	100%
regression_4709	315.930	100%
ram_blowup_regression	512.570	100%
rollup-root	498.200	100%
rollup-merge	472.940	100%
rollup-block-root	1220.000	100%
rollup-block-merge	498.210	100%
rollup-base-public	734.090	100%
rollup-base-private	590.400	100%
private-kernel-tail	180.820	100%
private-kernel-reset	245.440	100%
private-kernel-inner	208.840	100%

github-actions · 2025-01-15T18:27:20Z

Compilation Memory Report

Program	Peak Memory	%
keccak256	77.560	100%
workspace	123.850	100%
regression_4709	424.070	100%
ram_blowup_regression	1460.000	100%
rollup-root	601.170	100%
rollup-merge	494.140	100%
rollup-block-root-single-tx	16060.000	100%
rollup-block-root-empty	488.320	100%
rollup-block-root	16070.000	100%
rollup-block-merge	601.160	100%
rollup-base-public	2380.000	100%
rollup-base-private	1140.000	100%
private-kernel-tail	207.340	100%
private-kernel-reset	584.400	100%
private-kernel-inner	294.600	100%

github-actions

⚠️ Performance Alert ⚠️

Possible performance regression was detected for benchmark 'Compilation Time'.
Benchmark result of this commit is worse than the previous benchmark result exceeding threshold 1.10.

Benchmark suite	Current: `79f1d8e`	Previous: `1fb25cd`	Ratio
`sha256_regression`	`1.16` s	`1.01` s	`1.15`

This comment was automatically generated by workflow using github-action-benchmark.

CC: @TomAFrench

test_programs/parse_memory.sh

vezenovm · 2025-01-17T15:59:23Z

For a follow-up it would be nice to have separate pages for our memory/execution memory and compilation/execution timing reports. The dashboard is a bit long with all of our reports. I haven't looked into github-action-benchmark deeply though as to whether this is possible.

Co-authored-by: Maxim Vezenov <[email protected]>

TomAFrench · 2025-01-17T16:16:31Z

Yeah, I agree that it's not the prettiest but I don't want to invest too much time into it as I'm planning to migrate to Charlie's tool once it's ready.

github-actions · 2025-01-17T16:20:06Z

Changes to number of Brillig opcodes executed

Generated at commit: 7ac4bbe0ed1ae5af7ac1e886f4118136b9964231, compared to commit: ed12ad751336e726436b4fa7aad6958497e96285

🧾 Summary (10% most significant diffs)

Program	Brillig opcodes (+/-)	%
higher_order_functions_inliner_min	-165 ✅	-5.86%
higher_order_functions_inliner_zero	-90 ✅	-6.79%

Full diff report 👇

Program	Brillig opcodes (+/-)	%
slices_inliner_min	4,549 (-75)	-1.62%
higher_order_functions_inliner_min	2,651 (-165)	-5.86%
higher_order_functions_inliner_zero	1,236 (-90)	-6.79%

github-actions · 2025-01-17T16:20:08Z

Changes to Brillig bytecode sizes

Generated at commit: 7ac4bbe0ed1ae5af7ac1e886f4118136b9964231, compared to commit: ed12ad751336e726436b4fa7aad6958497e96285

🧾 Summary (10% most significant diffs)

Program	Brillig opcodes (+/-)	%
higher_order_functions_inliner_zero	+26 ❌	+3.90%
slices_inliner_min	-18 ✅	-0.68%

Full diff report 👇

Program	Brillig opcodes (+/-)	%
higher_order_functions_inliner_zero	692 (+26)	+3.90%
higher_order_functions_inliner_min	1,448 (+4)	+0.28%
slices_inliner_min	2,615 (-18)	-0.68%

…oir-lang/noir#7100) feat: Parser and formatter support for `enum`s (noir-lang/noir#7110) feat(brillig): SSA globals code gen (noir-lang/noir#7021) feat: `loop` keyword in runtime and comptime code (noir-lang/noir#7096) chore: Add benchmarking dashboard (noir-lang/noir#7068) feat(experimental): try to infer lambda argument types inside calls (noir-lang/noir#7088) feat(ssa): Add flag to DIE pass to be able to keep `store` instructions (noir-lang/noir#7106) chore: Cookbook Onboard integration (noir-lang/noir#7044) chore: lock to ubuntu 22.04 (noir-lang/noir#7098) fix: Remove unused brillig functions (noir-lang/noir#7102) chore(ssa): Use correct prefix when printing array values in global space (noir-lang/noir#7095)

…oir#7100) feat: Parser and formatter support for `enum`s (noir-lang/noir#7110) feat(brillig): SSA globals code gen (noir-lang/noir#7021) feat: `loop` keyword in runtime and comptime code (noir-lang/noir#7096) chore: Add benchmarking dashboard (noir-lang/noir#7068) feat(experimental): try to infer lambda argument types inside calls (noir-lang/noir#7088) feat(ssa): Add flag to DIE pass to be able to keep `store` instructions (noir-lang/noir#7106) chore: Cookbook Onboard integration (noir-lang/noir#7044) chore: lock to ubuntu 22.04 (noir-lang/noir#7098) fix: Remove unused brillig functions (noir-lang/noir#7102) chore(ssa): Use correct prefix when printing array values in global space (noir-lang/noir#7095)

…oir-lang/noir#7100) feat: Parser and formatter support for `enum`s (noir-lang/noir#7110) feat(brillig): SSA globals code gen (noir-lang/noir#7021) feat: `loop` keyword in runtime and comptime code (noir-lang/noir#7096) chore: Add benchmarking dashboard (noir-lang/noir#7068) feat(experimental): try to infer lambda argument types inside calls (noir-lang/noir#7088) feat(ssa): Add flag to DIE pass to be able to keep `store` instructions (noir-lang/noir#7106) chore: Cookbook Onboard integration (noir-lang/noir#7044) chore: lock to ubuntu 22.04 (noir-lang/noir#7098) fix: Remove unused brillig functions (noir-lang/noir#7102) chore(ssa): Use correct prefix when printing array values in global space (noir-lang/noir#7095)

…oir#7100) feat: Parser and formatter support for `enum`s (noir-lang/noir#7110) feat(brillig): SSA globals code gen (noir-lang/noir#7021) feat: `loop` keyword in runtime and comptime code (noir-lang/noir#7096) chore: Add benchmarking dashboard (noir-lang/noir#7068) feat(experimental): try to infer lambda argument types inside calls (noir-lang/noir#7088) feat(ssa): Add flag to DIE pass to be able to keep `store` instructions (noir-lang/noir#7106) chore: Cookbook Onboard integration (noir-lang/noir#7044) chore: lock to ubuntu 22.04 (noir-lang/noir#7098) fix: Remove unused brillig functions (noir-lang/noir#7102) chore(ssa): Use correct prefix when printing array values in global space (noir-lang/noir#7095)

TomAFrench and others added 2 commits January 14, 2025 23:27

chore: experimenting with benchmark action

a859398

Merge branch 'master' into tf/benchmark-action

4cd232e

TomAFrench added 2 commits January 15, 2025 17:05

.

7a2498a

.

e3020e3

TomAFrench and others added 7 commits January 15, 2025 18:29

.

4e2658f

Merge branch 'master' into tf/benchmark-action

5834839

Merge branch 'master' into tf/benchmark-action

99c0a80

Update reports.yml

d67ef8c

.

3d44973

.

781972a

.

69cc946

github-actions bot reviewed Jan 17, 2025

View reviewed changes

TomAFrench and others added 3 commits January 17, 2025 11:34

Update reports.yml

e66444a

Merge branch 'master' into tf/benchmark-action

1fb25cd

.

1b184b1

TomAFrench marked this pull request as ready for review January 17, 2025 15:08

TomAFrench requested a review from vezenovm January 17, 2025 15:11

TomAFrench changed the title ~~chore: experimenting with benchmark action~~ chore: Add benchmarking dashboard Jan 17, 2025

vezenovm reviewed Jan 17, 2025

View reviewed changes

test_programs/parse_memory.sh Outdated Show resolved Hide resolved

vezenovm approved these changes Jan 17, 2025

View reviewed changes

test_programs/parse_memory.sh Show resolved Hide resolved

Update test_programs/parse_memory.sh

79f1d8e

Co-authored-by: Maxim Vezenov <[email protected]>

TomAFrench enabled auto-merge January 17, 2025 16:36

TomAFrench added this pull request to the merge queue Jan 17, 2025

Merged via the queue into master with commit df71bde Jan 17, 2025
99 checks passed

TomAFrench deleted the tf/benchmark-action branch January 17, 2025 16:46

TomAFrench mentioned this pull request Jan 17, 2025

Remove comment system for time/memory benchmarks #7111

Open

AztecBot mentioned this pull request Jan 18, 2025

feat: Sync from noir AztecProtocol/aztec-packages#11294

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

chore: Add benchmarking dashboard #7068

chore: Add benchmarking dashboard #7068

TomAFrench commented Jan 14, 2025 •

edited

Loading

github-actions bot commented Jan 15, 2025 •

edited

Loading

github-actions bot commented Jan 15, 2025 •

edited

Loading

github-actions bot commented Jan 15, 2025 •

edited

Loading

github-actions bot commented Jan 15, 2025 •

edited

Loading

github-actions bot left a comment •

edited

Loading

vezenovm commented Jan 17, 2025

TomAFrench commented Jan 17, 2025

github-actions bot commented Jan 17, 2025

github-actions bot commented Jan 17, 2025

chore: Add benchmarking dashboard #7068

chore: Add benchmarking dashboard #7068

Conversation

TomAFrench commented Jan 14, 2025 • edited Loading

Description

Problem*

Summary*

Additional Context

Documentation*

PR Checklist*

github-actions bot commented Jan 15, 2025 • edited Loading

Compilation Report

github-actions bot commented Jan 15, 2025 • edited Loading

Execution Report

github-actions bot commented Jan 15, 2025 • edited Loading

Execution Memory Report

github-actions bot commented Jan 15, 2025 • edited Loading

Compilation Memory Report

github-actions bot left a comment • edited Loading

Choose a reason for hiding this comment

⚠️ Performance Alert ⚠️

vezenovm commented Jan 17, 2025

TomAFrench commented Jan 17, 2025

github-actions bot commented Jan 17, 2025

Changes to number of Brillig opcodes executed

🧾 Summary (10% most significant diffs)

github-actions bot commented Jan 17, 2025

Changes to Brillig bytecode sizes

🧾 Summary (10% most significant diffs)

TomAFrench commented Jan 14, 2025 •

edited

Loading

github-actions bot commented Jan 15, 2025 •

edited

Loading

github-actions bot commented Jan 15, 2025 •

edited

Loading

github-actions bot commented Jan 15, 2025 •

edited

Loading

github-actions bot commented Jan 15, 2025 •

edited

Loading

github-actions bot left a comment •

edited

Loading