Practice active benchmarking

[franz1981]

A common problem while running benchmarks is to assume what's the limiting factor the benchmark is stressing and the simpler the benchmark, the easier is to believe to know the unknown/ignore there are any.
To fix this, is highly suggested to practice https://www.brendangregg.com/activebenchmarking.html in the form of opt-in profiling/monitoring for the different components involved, whilst the tests are running
eg

1. Enable lightweight monitoring, like sar/mpstat/time/pidstat or docker stats, against all components
2. Opt-in to make sure that with/without such observability tools, there won't be any unexpected and unpredictable "observer" effect

As part of this, although with an higher and more biased resolution level, I would enable (still opt-in because more heavy, for the type of test) startup profiling too, which I can help setup, for both JVM and native image mode.
The latter is not trivial, but afaik, be we can ask our Mandrel team if the CPU profiler recently introduced for native-image is suitable for this job, and accurate enough, compared to Linux perf. If we enable native image profiling this is going (iirc) to impact building native too, because it requires additional compilation flags to produce debug symbols required to resolve the application frame stacks properly.

In short:

1. let's start by proving lightweight monitoring (opt-in) via sar/mpstat/time/pidstat on components
2. Optional, but highly recommended, enable system wide, to collect similar statistics: this is very tied to our current choice to run everything local, but is good to ensure there is not overall saturation in the system due to scarse resources or noisy neighbors presence
3. Eventually, add finer grain profiling to JVM mode and native image (JVM already does it under load, but not at start-up, which is a whole different beast)

For 1, we can collect the output of monitoring and report it raw too; it doesn't need ATM to be part of the results, but still we need to make it easier to consume in order to detect unwanted/unexpected behaviors which could invalidate our results.

This is very key to make sure our results are valid, especially if run by others which environment is drastically different from our lab e.g. while they mine Bitcoin with their CPUs eheh

I will provide few pointers of existing qDup scrips from other OSS test we performed which could be (re)used here.

[franz1981]

@holly-cummins this applies, in some regard to your tests as well, and should be easy to show

e.g.

1. https://github.com/franz1981/quarkus-profiling-workshop/blob/master/scripts/benchmark.sh#L203-L214 it produce, under the load gen test, some reporting re how the server consume resources: you can do it as well for postgres via docker/podman stats
2. for startup test you could use time or others which are more suitable for short living cases

During a conference/presentation this can help to:

1. show that startup cost is more dominated by usr time than people would expect (If I'm right :P which could not be the case!)
2. during the load test instead (which we claim to be "intensive" and stressing the framework) we could show that it is actually more stressful for the SUT than the DBMS, making the "naive but not too naive" approach, still a decent test compared to any broken benchmark on internet (which is what we want to achieve right?)

[franz1981]

@edeandrea here we go:

https://github.com/RedHatPerf/superheroes-automation/blob/main/profiling.yaml

Many thanks to @lampajr for these 🙏

My take on some of these:

• for CPU-centric debugging → mpstat e.g mpstat -P ALL 1
• Overall system health → vmstat .e.g vmstat 1
• Finer grain misbehaviours → pidstat e.g. pidstat -u -p <server pid> 1
• startup wide metrics → time

Why I wouldn't trust docker/podman stats vs the DBMS instead:

• CPU stats are cgroup-aggregated
• I/O stats are incomplete
• it lies about memory
• it lies about latency specific events e.g. voluntary/involuntary context switches, blocking syscalls, etc etc

Which means I would use pidstat for postgres, but you need first to collect the "host" pid of the container
in the infra script (maybe?) with

POSTGRES_HOST_PID=$(docker inspect -f '{{.State.Pid}}' postgres)
pidstat -u -p $(pgrep -d, -P $POSTGRES_HOST_PID) 1

Optionally add -w -d for context switches / I/O.
Why with postgres using pidstat -u -p <host id> 1 is not enough?
AFAIK, but I'm no postgres expert, postgres is multi process (within the container itself) so just -p won't show the child processes activities (at all!! even if fully consume the cpu of the host/container!)

[holly-cummins]

I love the sound of this.

Mini-clarification, for my understanding. What do you mean by "this is very tied to our current choice to run everything local"? Is "local" ~= "bare-metal in the perf lab"?

[franz1981]

Is "local" ~= "bare-metal in the perf lab"?

Nope, but the fact that we run all 3 components (DBMS, server + load generator) in the same machine, instead of using separated machines.
It both affect the OS networking stack, cpu usage and power drawn (which would differ from a real world case where networking is happening for real) and clearly can introduce more easily unwanted "noisy neighbour" effects and interference between each others.

[brunobat]

For reference, on the OTel project we can see the active monitoring here (1 example): We have active monitoring in place: https://ci.app-svc-perf.corp.redhat.com/job/ExternalTeams/job/OTel/job/OTel-Quarkus-Observability-Perf/263/artifact/run/server/monitoring/

... and is configured here: https://github.com/brunobat/quarkus-observability-perf/blob/82d07a99632e45f7ff2fc82f959dde09ab882fc6/qDup.yaml#L350