Add Debug scripts #1839

JulianGCalderon · 2024-09-17T20:49:30Z

Add debug scripts

This PR was initially inside of #1824, but I moved it to reduce the size of that PR.

Description

It adds some scripts (and documents them) for debugging differences in memory, trace, public and private air inputs.

Checklist

Linked to Github Issue
Unit tests added
Integration tests added.
This change requires new documentation.
- Documentation has been added/updated.
- CHANGELOG has been updated.

github-actions · 2024-09-17T20:59:01Z

**Hyper Thereading Benchmark results**




hyperfine -r 2 -n "hyper_threading_main threads: 1" 'RAYON_NUM_THREADS=1 ./hyper_threading_main' -n "hyper_threading_pr threads: 1" 'RAYON_NUM_THREADS=1 ./hyper_threading_pr'
Benchmark 1: hyper_threading_main threads: 1
  Time (mean ± σ):     30.108 s ±  0.065 s    [User: 29.330 s, System: 0.776 s]
  Range (min … max):   30.062 s … 30.155 s    2 runs
 
Benchmark 2: hyper_threading_pr threads: 1
  Time (mean ± σ):     30.469 s ±  0.130 s    [User: 29.685 s, System: 0.782 s]
  Range (min … max):   30.377 s … 30.561 s    2 runs
 
Summary
  'hyper_threading_main threads: 1' ran
    1.01 ± 0.00 times faster than 'hyper_threading_pr threads: 1'




hyperfine -r 2 -n "hyper_threading_main threads: 2" 'RAYON_NUM_THREADS=2 ./hyper_threading_main' -n "hyper_threading_pr threads: 2" 'RAYON_NUM_THREADS=2 ./hyper_threading_pr'
Benchmark 1: hyper_threading_main threads: 2
  Time (mean ± σ):     16.780 s ±  0.023 s    [User: 29.555 s, System: 0.776 s]
  Range (min … max):   16.763 s … 16.796 s    2 runs
 
Benchmark 2: hyper_threading_pr threads: 2
  Time (mean ± σ):     16.912 s ±  0.075 s    [User: 29.881 s, System: 0.763 s]
  Range (min … max):   16.859 s … 16.966 s    2 runs
 
Summary
  'hyper_threading_main threads: 2' ran
    1.01 ± 0.00 times faster than 'hyper_threading_pr threads: 2'




hyperfine -r 2 -n "hyper_threading_main threads: 4" 'RAYON_NUM_THREADS=4 ./hyper_threading_main' -n "hyper_threading_pr threads: 4" 'RAYON_NUM_THREADS=4 ./hyper_threading_pr'
Benchmark 1: hyper_threading_main threads: 4
  Time (mean ± σ):     11.444 s ±  0.184 s    [User: 41.568 s, System: 0.977 s]
  Range (min … max):   11.313 s … 11.574 s    2 runs
 
Benchmark 2: hyper_threading_pr threads: 4
  Time (mean ± σ):     11.502 s ±  0.688 s    [User: 41.806 s, System: 0.968 s]
  Range (min … max):   11.015 s … 11.988 s    2 runs
 
Summary
  'hyper_threading_main threads: 4' ran
    1.01 ± 0.06 times faster than 'hyper_threading_pr threads: 4'




hyperfine -r 2 -n "hyper_threading_main threads: 6" 'RAYON_NUM_THREADS=6 ./hyper_threading_main' -n "hyper_threading_pr threads: 6" 'RAYON_NUM_THREADS=6 ./hyper_threading_pr'
Benchmark 1: hyper_threading_main threads: 6
  Time (mean ± σ):     11.369 s ±  0.026 s    [User: 41.853 s, System: 0.951 s]
  Range (min … max):   11.351 s … 11.387 s    2 runs
 
Benchmark 2: hyper_threading_pr threads: 6
  Time (mean ± σ):     11.513 s ±  0.475 s    [User: 41.796 s, System: 0.999 s]
  Range (min … max):   11.176 s … 11.849 s    2 runs
 
Summary
  'hyper_threading_main threads: 6' ran
    1.01 ± 0.04 times faster than 'hyper_threading_pr threads: 6'




hyperfine -r 2 -n "hyper_threading_main threads: 8" 'RAYON_NUM_THREADS=8 ./hyper_threading_main' -n "hyper_threading_pr threads: 8" 'RAYON_NUM_THREADS=8 ./hyper_threading_pr'
Benchmark 1: hyper_threading_main threads: 8
  Time (mean ± σ):     11.237 s ±  0.159 s    [User: 41.822 s, System: 0.979 s]
  Range (min … max):   11.125 s … 11.349 s    2 runs
 
Benchmark 2: hyper_threading_pr threads: 8
  Time (mean ± σ):     11.364 s ±  0.032 s    [User: 41.818 s, System: 1.002 s]
  Range (min … max):   11.341 s … 11.387 s    2 runs
 
Summary
  'hyper_threading_main threads: 8' ran
    1.01 ± 0.01 times faster than 'hyper_threading_pr threads: 8'




hyperfine -r 2 -n "hyper_threading_main threads: 16" 'RAYON_NUM_THREADS=16 ./hyper_threading_main' -n "hyper_threading_pr threads: 16" 'RAYON_NUM_THREADS=16 ./hyper_threading_pr'
Benchmark 1: hyper_threading_main threads: 16
  Time (mean ± σ):     11.173 s ±  0.198 s    [User: 42.185 s, System: 1.092 s]
  Range (min … max):   11.033 s … 11.313 s    2 runs
 
Benchmark 2: hyper_threading_pr threads: 16
  Time (mean ± σ):     11.411 s ±  0.043 s    [User: 42.031 s, System: 1.083 s]
  Range (min … max):   11.381 s … 11.442 s    2 runs
 
Summary
  'hyper_threading_main threads: 16' ran
    1.02 ± 0.02 times faster than 'hyper_threading_pr threads: 16'

github-actions · 2024-09-17T21:11:45Z

Benchmark Results for unmodified programs 🚀

Command	Mean [s]	Min [s]	Max [s]	Relative
`base big_factorial`	2.392 ± 0.014	2.376	2.416	1.00
`head big_factorial`	2.401 ± 0.012	2.382	2.429	1.00 ± 0.01

Command	Mean [s]	Min [s]	Max [s]	Relative
`base big_fibonacci`	2.361 ± 0.062	2.313	2.515	1.01 ± 0.03
`head big_fibonacci`	2.332 ± 0.021	2.310	2.384	1.00

Command	Mean [s]	Min [s]	Max [s]	Relative
`base blake2s_integration_benchmark`	8.626 ± 0.190	8.436	8.934	1.00
`head blake2s_integration_benchmark`	8.810 ± 0.322	8.482	9.491	1.02 ± 0.04

Command	Mean [s]	Min [s]	Max [s]	Relative
`base compare_arrays_200000`	2.525 ± 0.063	2.473	2.669	1.00 ± 0.04
`head compare_arrays_200000`	2.524 ± 0.066	2.472	2.686	1.00

Command	Mean [s]	Min [s]	Max [s]	Relative
`base dict_integration_benchmark`	1.652 ± 0.006	1.644	1.661	1.01 ± 0.01
`head dict_integration_benchmark`	1.634 ± 0.008	1.628	1.653	1.00

Command	Mean [s]	Min [s]	Max [s]	Relative
`base field_arithmetic_get_square_benchmark`	1.396 ± 0.036	1.366	1.469	1.02 ± 0.03
`head field_arithmetic_get_square_benchmark`	1.368 ± 0.010	1.353	1.388	1.00

Command	Mean [s]	Min [s]	Max [s]	Relative
`base integration_builtins`	8.932 ± 0.501	8.548	10.289	1.01 ± 0.07
`head integration_builtins`	8.831 ± 0.287	8.522	9.515	1.00

Command	Mean [s]	Min [s]	Max [s]	Relative
`base keccak_integration_benchmark`	9.065 ± 0.206	8.810	9.470	1.00
`head keccak_integration_benchmark`	9.126 ± 0.381	8.771	10.131	1.01 ± 0.05

Command	Mean [s]	Min [s]	Max [s]	Relative
`base linear_search`	2.500 ± 0.041	2.465	2.590	1.00
`head linear_search`	2.527 ± 0.077	2.463	2.704	1.01 ± 0.04

Command	Mean [s]	Min [s]	Max [s]	Relative
`base math_cmp_and_pow_integration_benchmark`	1.713 ± 0.020	1.696	1.767	1.00
`head math_cmp_and_pow_integration_benchmark`	1.730 ± 0.061	1.697	1.866	1.01 ± 0.04

Command	Mean [s]	Min [s]	Max [s]	Relative
`base math_integration_benchmark`	1.671 ± 0.019	1.657	1.725	1.01 ± 0.01
`head math_integration_benchmark`	1.661 ± 0.015	1.647	1.689	1.00

Command	Mean [s]	Min [s]	Max [s]	Relative
`base memory_integration_benchmark`	1.400 ± 0.014	1.389	1.436	1.00
`head memory_integration_benchmark`	1.402 ± 0.013	1.388	1.429	1.00 ± 0.01

Command	Mean [s]	Min [s]	Max [s]	Relative
`base operations_with_data_structures_benchmarks`	1.780 ± 0.030	1.760	1.860	1.01 ± 0.02
`head operations_with_data_structures_benchmarks`	1.769 ± 0.012	1.748	1.793	1.00

Command	Mean [ms]	Min [ms]	Max [ms]	Relative
`base pedersen`	571.3 ± 2.6	566.9	576.1	1.00 ± 0.00
`head pedersen`	570.8 ± 1.2	569.3	573.0	1.00

Command	Mean [ms]	Min [ms]	Max [ms]	Relative
`base poseidon_integration_benchmark`	674.9 ± 6.7	665.4	685.9	1.00 ± 0.01
`head poseidon_integration_benchmark`	673.6 ± 5.9	665.2	684.6	1.00

Command	Mean [s]	Min [s]	Max [s]	Relative
`base secp_integration_benchmark`	1.984 ± 0.014	1.968	2.018	1.00
`head secp_integration_benchmark`	1.999 ± 0.036	1.973	2.095	1.01 ± 0.02

Command	Mean [ms]	Min [ms]	Max [ms]	Relative
`base set_integration_benchmark`	683.6 ± 5.8	677.8	693.2	1.00 ± 0.02
`head set_integration_benchmark`	683.1 ± 11.6	676.2	714.6	1.00

Command	Mean [s]	Min [s]	Max [s]	Relative
`base uint256_integration_benchmark`	4.808 ± 0.112	4.698	5.076	1.00
`head uint256_integration_benchmark`	4.818 ± 0.108	4.716	5.081	1.00 ± 0.03

codecov · 2024-09-17T21:12:22Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 94.83%. Comparing base (5f8a011) to head (6f292f3).
Report is 1 commits behind head on main.

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #1839   +/-   ##
=======================================
  Coverage   94.83%   94.83%           
=======================================
  Files         101      101           
  Lines       39579    39579           
=======================================
  Hits        37536    37536           
  Misses       2043     2043

Flag	Coverage Δ
	`94.83% <ø> (?)`

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

pefontana · 2024-09-20T21:20:27Z

CHANGELOG.md

@@ -2,6 +2,8 @@

 #### Upcoming Changes

+* docs: Add debugging scripts and documentation [#1839](https://github.com/lambdaclass/cairo-vm/pull/1839)


Documentation and this kind of scripts that don't modify the VM doesn't need a line in the changelog
Remove it

Done. I added "documentation" label to ignore the changelog check.

pefontana · 2024-09-20T21:23:35Z

docs/debugging.md

+
+To compare the public inputs, run:
+```bash
+scripts/air_public_inputs_differ.bash file1 file2


Can you be a little clearer here.
Instead of file do <AIR-PUBLIC-INPUT-1> <AIR-PUBLIC-INPUT-2>

Try to do the same to the other commands

* Add debug scripts * Add debugging document * Add changelog * Revert changelog * Improve script usage code blocks * Remove extra space

* Fix Zero segment location. * Fixed has_zero_segment naming * Fix prover input. * Fixed version reading when no version is supplied * Added change to changelog. * fix test_from_serializable() * fix panic_impl error * fix cairo version * Add dummy changelog * Pin wasm-bindgen * Register change in CHANGELOG * Update Cargo.lock * Remove changes from CHANGELOG * Add argument parsing for layout params file * Add dynamic support (no implement) * Add cairo_layout_params_file.example.json * Implement dynamic layout creation * Update CHANGELOG * Add cli dynamic support for cairo 1 * Make wasm compatible * Use public_memory_fraction = 4 vy default * Deserialize bool from int * Add comparison with python-vm (failing) * Rebuild .rs files in makefile * Use 8 as dynamic public_memory_fraction The same value is used in python-vm * Use None ratio for dynamic unused builtins * Add rangecheck96 to private inputs * Make dyn py files depend on params_file * Use cpu_component_step=1 by default * Fix typo in private inputs * Add range check value to air private input test * Fix zero segment location * Use zero builtin instead of None * Add debug scripts * Remove dup makefile recipes * remove outdated test * Enable ensure-no_std on test * Fix tests * Add correct test * Rename tset * Add comment * Add debugging document * Update cairo layout params file * Remove duplicated range check * Remove dup * Add support for dynamic memory units per step * Add changelog * Add support for negative log units per step * Add LowRatio struct to RangeCheck and Mod builtins * Fix test * Use ratio_den in get_allocated_instances * Remove debugging and scrippts (moveed to another branch) * Add comment * Add tests * Add dynamic test to cairo-vm-cli * Add parse test * Remove compare all dynamic * Add script for comparing with dynamic layouts * Add tests to workflow * Delete logic changes They are going to be moved to another branch * Delete more logic changes * Reorder fields * Remove ref * Add modulo tests with all_cairo * Add tests * Update rust.yml * Rename compare_outputs_dynamic_layout.sh to compare_outputs_dynamic_layouts.sh * Update test script * Remove tests * Enforce prover constraints in add, sub, and mul * Remove debug prints * Add tests * Update CHANGELOG.md * Fix serialization * Comment failing test * Uncomment test * Add fractional units per step test * Change safe_div for div_ceil (bug) * Add ratio_den tests * Fix tests * Remove operation struct and use builtin type instead * Add unit tests to modulo builtin operations * Fix security error message * Add mod_builtin to coverage * Test custom serde impl * Upload mod_builtin coverage * Test custom serde impl * Upload mod_builtin coverage * Add Debug scripts (#1839) * Add debug scripts * Add debugging document * Add changelog * Revert changelog * Improve script usage code blocks * Remove extra space * Add cpu_component_step * Add comments * Add cairo pie tests * Restore zip * Use .rs.pie.zip * Update Changelog * Add dynamic layout documentation * Update CHANGELOG.md --------- Co-authored-by: Alon Titelman <[email protected]> Co-authored-by: Yuval Goldberg <[email protected]> Co-authored-by: Omri Eshhar <[email protected]> Co-authored-by: Pedro Fontana <[email protected]>

JulianGCalderon added 3 commits September 17, 2024 17:44

Add debug scripts

6f98945

Add debugging document

48c3438

Add changelog

cdcefc9

JulianGCalderon marked this pull request as ready for review September 19, 2024 21:04

JulianGCalderon requested review from igaray, Oppen, fmoletta, juanbono and pefontana as code owners September 19, 2024 21:04

pefontana reviewed Sep 20, 2024

View reviewed changes

JulianGCalderon added 2 commits September 20, 2024 18:52

Revert changelog

0bb3f07

Improve script usage code blocks

ef97fc1

JulianGCalderon added the documentation Improvements or additions to documentation label Sep 20, 2024

Remove extra space

6f292f3

Oppen approved these changes Sep 23, 2024

View reviewed changes

pefontana approved these changes Sep 23, 2024

View reviewed changes

pefontana added this pull request to the merge queue Sep 23, 2024

Merged via the queue into main with commit 63b4a55 Sep 23, 2024
72 checks passed

pefontana deleted the debug-scripts branch September 23, 2024 15:47

JulianGCalderon added a commit that referenced this pull request Sep 23, 2024

Add Debug scripts (#1839)

ba44310

* Add debug scripts * Add debugging document * Add changelog * Revert changelog * Improve script usage code blocks * Remove extra space

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add Debug scripts #1839

Add Debug scripts #1839

JulianGCalderon commented Sep 17, 2024 •

edited

Loading

github-actions bot commented Sep 17, 2024 •

edited

Loading

github-actions bot commented Sep 17, 2024 •

edited

Loading

codecov bot commented Sep 17, 2024 •

edited

Loading

pefontana Sep 20, 2024

JulianGCalderon Sep 20, 2024

pefontana Sep 20, 2024

JulianGCalderon Sep 20, 2024

		@@ -2,6 +2,8 @@

		#### Upcoming Changes

		* docs: Add debugging scripts and documentation [#1839](https://github.com/lambdaclass/cairo-vm/pull/1839)

Add Debug scripts #1839

Add Debug scripts #1839

Conversation

JulianGCalderon commented Sep 17, 2024 • edited Loading

Add debug scripts

Description

Checklist

github-actions bot commented Sep 17, 2024 • edited Loading

github-actions bot commented Sep 17, 2024 • edited Loading

codecov bot commented Sep 17, 2024 • edited Loading

Codecov Report

pefontana Sep 20, 2024

Choose a reason for hiding this comment

JulianGCalderon Sep 20, 2024

Choose a reason for hiding this comment

pefontana Sep 20, 2024

Choose a reason for hiding this comment

JulianGCalderon Sep 20, 2024

Choose a reason for hiding this comment

JulianGCalderon commented Sep 17, 2024 •

edited

Loading

github-actions bot commented Sep 17, 2024 •

edited

Loading

github-actions bot commented Sep 17, 2024 •

edited

Loading

codecov bot commented Sep 17, 2024 •

edited

Loading