Llama 3.2 3B Instruct Q40 Benchmarks, 1x, 2x, 4x, 8x Raspberry Pi 4B 8GB

[githuba9f5404]

Running on this hardware setup, overclocked with a clock speed of 2000 MHZ (standard is 1500 MHZ). All benchmarks run with --steps 64 and --prompt "The Eiffel Tower is". Output was all reasonable. Model converted to q40, run with q80 buffer.

1 x Node

Generated tokens:    64
Avg tokens / second: 1.37
Avg generation time: 728.80 ms
Avg inference time:  723.45 ms
Avg transfer time:   3.14 ms

2 x Nodes

Generated tokens:    64
Avg tokens / second: 2.04
Avg generation time: 489.94 ms
Avg inference time:  462.66 ms
Avg transfer time:   25.16 ms

4 x Nodes

Generated tokens:    64
Avg tokens / second: 2.68
Avg generation time: 373.38 ms
Avg inference time:  347.77 ms
Avg transfer time:   23.55 ms

8 x Nodes

Generated tokens:    64
Avg tokens / second: 3.22
Avg generation time: 310.47 ms
Avg inference time:  287.23 ms
Avg transfer time:   21.16 ms