Skip to content

Conversation

@johannes94
Copy link

@johannes94 johannes94 commented Oct 8, 2025

Hello there @bep.

One of the projects I'm working on makes heavy use of the hashstructure lib.

Since we're hashing rather big objects, I noticed huge allocation overhead in the library and did some optimizations.
In particular:

  • Avoid using reflect.ValueOf for obvious string values
  • Avoid allocating visitOps object in a loop
  • Changes binary.Write to direct byte operations using a shared buffer for number types
  • Use the unsafe pkg for zero copy binary writes for strings
  • Only wrapping reflect.Value objects with v.Int(), v.Interface() etc. if absolutely necessary

I'm not sure if this is a fork you intent to maintain as a library for others to use, but since I'm building on your previous improvements I thought this might be valuable to you.

Side Note: Those improvements where partly suggested by AI, but I carefully reviewed them and made sure objects we hash within our projects produce hashes consistent with the previous version.

Benchmark:

goos: darwin
goarch: arm64
pkg: github.com/gohugoio/hashstructure
cpu: Apple M4 Pro
                  │   old.txt    │               new.txt                │
                  │    sec/op    │    sec/op     vs base                │
Map-14              3.862µ ± 24%   2.752µ ± 19%  -28.74% (p=0.000 n=10)
String/default-14   63.02n ±  5%   48.54n ±  4%  -22.98% (p=0.000 n=10)
String/xxhash-14    33.85n ±  7%   30.54n ±  9%   -9.79% (p=0.007 n=10)
geomean             202.0n         159.8n        -20.89%

                  │   old.txt   │               new.txt                │
                  │    B/op     │    B/op     vs base                  │
Map-14              1113.0 ± 0%   665.0 ± 0%  -40.25% (p=0.000 n=10)
String/default-14    56.00 ± 0%   24.00 ± 0%  -57.14% (p=0.000 n=10)
String/xxhash-14     16.00 ± 0%   16.00 ± 0%        ~ (p=1.000 n=10) ¹
geomean              99.91        63.44       -36.50%
¹ all samples are equal

                  │   old.txt   │               new.txt                │
                  │  allocs/op  │ allocs/op   vs base                  │
Map-14              120.00 ± 0%   43.00 ± 0%  -64.17% (p=0.000 n=10)
String/default-14    3.000 ± 0%   2.000 ± 0%  -33.33% (p=0.000 n=10)
String/xxhash-14     1.000 ± 0%   1.000 ± 0%        ~ (p=1.000 n=10) ¹
geomean              7.114        4.414       -37.95%
¹ all samples are equal

@bep
Copy link
Member

bep commented Oct 9, 2025

Thanks for this, much appreciated. And the numbers looks even better on my MacBook Pro M1:

cpu: Apple M1 Pro
                  │ master.bench │ jmalsam-upstream-performance-optimizations.bench │
                  │    sec/op    │          sec/op            vs base               │
Map-10              3.831µ ± ∞ ¹                2.315µ ± ∞ ¹  -39.57% (p=0.029 n=4)
String/default-10   55.92n ± ∞ ¹                30.50n ± ∞ ¹  -45.45% (p=0.029 n=4)
String/xxhash-10    17.12n ± ∞ ¹                13.93n ± ∞ ¹  -18.63% (p=0.029 n=4)
geomean             154.2n                      99.45n        -35.51%
¹ need >= 6 samples for confidence interval at level 0.95

                  │ master.bench │ jmalsam-upstream-performance-optimizations.bench │
                  │     B/op     │          B/op            vs base                 │
Map-10              1080.0 ± ∞ ¹               632.0 ± ∞ ¹  -41.48% (p=0.029 n=4)
String/default-10   40.000 ± ∞ ¹               8.000 ± ∞ ¹  -80.00% (p=0.029 n=4)
String/xxhash-10     0.000 ± ∞ ¹               0.000 ± ∞ ¹        ~ (p=1.000 n=4) ²
geomean                        ³                            -51.09%               ³
¹ need >= 6 samples for confidence interval at level 0.95
² all samples are equal
³ summaries must be >0 to compute geomean

                  │ master.bench │ jmalsam-upstream-performance-optimizations.bench │
                  │  allocs/op   │        allocs/op         vs base                 │
Map-10              116.00 ± ∞ ¹               39.00 ± ∞ ¹  -66.38% (p=0.029 n=4)
String/default-10    2.000 ± ∞ ¹               1.000 ± ∞ ¹  -50.00% (p=0.029 n=4)
String/xxhash-10     0.000 ± ∞ ¹               0.000 ± ∞ ¹        ~ (p=1.000 n=4) ²

@bep bep merged commit 1f6d247 into gohugoio:master Oct 9, 2025
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants