Install the compilers needed to build pandas in the system:
apt install gcc g++
Create a user to run the benchmarks, and clone this repository in its home.
Install pixi, which we use to manage the environment that runs asv. Note that the the environment to run the benchmarks is managed by asv and it is different from the pixi environment:
curl -fsSL https://pixi.sh/install.sh | bash
Clone the pandas repository inside the pandas-benchmarks
directory:
cd pandas-benchmarks
git clone https://github.com/pandas-dev/pandas.git
We use pixi to manage the environment and run the benchmarks:
pixi run bench
We may want to implement a script that runs benchmarks continually (a new run starts when the previous finishes, indefinetly). But for now we are using cron.
To set up cron to run the benchmarks automatically we can use:
0 */3 * * * cd pandas-benchmarks && /home/bench/.pixi/bin/pixi run bench >> bench.log 2>&1
Note that the frequency should avoid starting a new job when the previous has not finished, so if the benchmarks take 2.5 hours to complete, we should schedule the runs to for example every 3 hours.
To view the log of cron executions we can run:
grep CRON /var/log/syslog | grep "(bench)"
Everything that happens in the system while running the benchmarks causes an impact, meaning that benchmarks will run faster when there is not much noise, and will run slower when there is. For example, if the core running the benchmarks takes care of an operating system interruption, this will cause a context switch, will flush the CPU caches, and the benchmark will take longer. Even if every benchmark is run multiple times, this variance makes our results worse and likely to cause false positives. This section is about trying to make the system more stable and reduce the variance of the execution time of benchmarks.
First thing we can do is to isolate the CPUs where the benchmarks run. This means that the operating system won't use the CPU unless a process is explicitly started with a CPU affinity to that core.
First, to check the cores available in the system we can run:
$ lscpu --all --extended
CPU NODE SOCKET CORE L1d:L1i:L2:L3 ONLINE MAXMHZ MINMHZ MHZ
0 0 0 0 0:0:0:0 yes 4900.0000 800.0000 4798.3130
1 0 0 1 1:1:1:0 yes 4900.0000 800.0000 4603.2891
2 0 0 2 2:2:2:0 yes 4900.0000 800.0000 4000.0000
3 0 0 3 3:3:3:0 yes 4900.0000 800.0000 4000.0000
4 0 0 0 0:0:0:0 yes 4900.0000 800.0000 4000.0000
5 0 0 1 1:1:1:0 yes 4900.0000 800.0000 4000.0000
6 0 0 2 2:2:2:0 yes 4900.0000 800.0000 4782.7388
7 0 0 3 3:3:3:0 yes 4900.0000 800.0000 4000.0000
The CPU
column shows that the benchmarks server has 8 cores, and the CORE
column shows that those are using 4 different physical cores (every physical
core is used by two separate pipelines or virtual cores, referred by Intel
as hyperthreads). We need to isolate physical cores, so the OS does not
execute anything in the other pipeline either, which would also slow down
the benchmark execution.
To isolate CPUs we need to add parameters to the kernel. To do so, we edit
the file /etc/default/grub
and do these changes:
# Find this line:
GRUB_CMDLINE_LINUX_DEFAULT="quiet splash"
# Replace it with this line (add the parameters at the end):
GRUB_CMDLINE_LINUX_DEFAULT="quiet splash isolcpus=3,7 nohz_full=3,7"
This will isolate the physical core 3, via its two virtual cores 3 and 7. It will also remove these cores from the operating system scheduler ticks. We can surely isolate more cores, for now we just start by one for simplicity.
For the changes to have an effect we first need to update the actual grub
configuration with the changes in /etc/default/grub.d/50-cloudimg-settings.cfg
.
In general /etc/default/grub
is used for grub settings, but OVH overwrites the
content of that file with 50-cloudimg-settings.cfg
. Note that grub does not read
directly from those files, so it is needed to execute update-grub
or grub-mkconfig
which parse these files and write to /boot/grub.grub.cfg
which is the one used by
the operating system. After executing one of those commands it is needed to restart
the system so the running kernel contains the new parameters. In practice this is as
simple as tuning the next commands
$ sudo vim /etc/default/grub.d/50-cloudimg-settings.cfg # and make changes above
$ sudo update-grub
$ sudo reboot
Once the system is restarted we should check that the CPUs are indeed isolated as expected. This can be done checking the information in the next files:
$ cat /sys/devices/system/cpu/isolated
3,7
We can also see that the operating system is not running tasks in the isolated CPUs by generating process and checking CPU usage with htop:
$ apt install stress
$ stress --cpu 8
$ htop # in a different terminal
Isolation works for processes running in the user space, but not in the system space. Ideally, we would like to avoid interruptions running in our isolated kernel. While this is a complex topic, and not all intererruptions can run in any core, to limit the number of cores every interruption runs in a general way, this command can be used:
for IRQ_AFFINITY_FILE in $(find . -name smp_affinity); do echo 77 | sudo tee $IRQ_AFFINITY_FILE; done
Note that for some interruptions the command will fail. Also note that 77
is a binary
mask in hexadecimal representing 0111 0111
(4th and 8th CPUs are not allowed to run the
interruption).
Modern CPUs are able to scale their frequency depending on work load or temperature. When a CPU is idle it will decrease its frequency to save energy. Also, when a CPU is busy and its temperature increases, it will eventually decrease its frequency so the temperature goes back to safe level.
Most of these frequency scaling technologies can be disabled via the system BIOS, but we do not have control of it in the servers in a data center, and disabling them may make frequency slow, and the benchmark suite take much longer to run (something like double the time based on past tests).
There are some things we have control of at runtime. We should be able to disable TurboBoost via:
echo 1 | sudo tee /sys/devices/system/cpu/intel_pstate/no_turbo
We can also install cpufreq
which gives informations and allow to control certain features with:
sudo apt install linux-tools-generic
While the system introduces noise to due to CPU scaling or our benchmark process being interrupted by other processes and interruptions, there are other sources of noise that cause variance in the results of our benchmarks.
The main ones identifies are:
- I/O operations
- Unpredictable CPU cache misses
- Randomness (for example, our benchmarks on functions that check duplicates are affected by the randomness in the hashing functions for the used hash tables).