Create hyperfoil yaml rather than running wrk directly?

[edeandrea]

@franz1981 when we do the warmup & the test

spring-quarkus-perf-comparison/scripts/perf-lab/main.yml

Lines 365 to 371 in d8a009d

	- sh: sleep ${{PAUSE_TIME}}
	- sh: ${{RUN.WRK_BIN}} -t 16 -c 100 -d 120 ${{TARGET_URL}} 2>&1 >${{REPO_DIR}}/logs/wrk-warmup-${{RUNTIME.name}}-${{ITERATION}}.log
	- regex: unable to connect
	then:
	- abort: unable to connect to target application
	- signal: LOAD_STEADY_STATE_START
	- sh: ${{RUN.WRK_BIN}} -t 16 -c 100 -d 30 ${{TARGET_URL}} 2>&1 >${{REPO_DIR}}/logs/wrk-${{RUNTIME.name}}-${{ITERATION}}.log

Would it be better to create a hyperfoil yaml file that has both stages and run jbang run@hyperfoil instead?

[franz1981]

It enables better telemetry and control but depends if we need such additional control here.
It would still require an additional parsing effort (not a biggie but still...) to obtain TPS and other metrics.
If you want is fairly easy to write one which mimic both wrk/wrk2

[edeandrea]

@franz1981 I've started working on this and came up with something. When I tried running it with jbang run@hyperfoil I noticed that all I got was the HTML. I had to go digging through log files to find the all.json, but even the all.json didn't have the throughput numbers.

This is what the script currently does:

# warmup
jbang wrk@hyperfoil -t2 -c100 -d2m http://localhost:8080/fruits

# Load test
jbang wrk@hyperfoil -t2 -c100 -d30s http://localhost:8080/fruits

When I ran that on my test box I get

Running 30s test @ http://localhost:8080/fruits
  2 threads and 100 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    10.30ms    9.33ms 574.62ms   91.97%
    Req/Sec   9391.87   2015.05  10361.00     93.55
  291148 requests in 30.012s,   1.52GB read
Requests/sec: 9701.05
Transfer/sec:  51.91MB

For hyperfoil, I built this:

name: app-load-test
http:
  - protocol: !param PROTOCOL http
    host: !param HOST localhost
    port: !param PORT 8080
    sharedConnections: !param SHARED_CONNECTIONS 100

phases:
  - loadTest:
      always:
        users: !param LOAD_USERS 2
        duration: !param LOAD_DURATION 30s
        startAfter: warmup
        scenario:
          - executeAction: &executeAction
              - httpRequest:
                  GET: !param PATH /fruits
                  headers:
                    accept: application/json

  - warmup:
      always:
        users: !param WARMUP_USERS 2
        duration: !param WARMUP_DURATION 2m
        scenario:
          - executeAction: *executeAction

Then I run it with jbang run@hyperfoil app-load-test.hf.yml. Once it comples I can dig and find all.json. Inside all.json I can find

"stats": [
    {
      "name": "loadTest",
      "phase": "loadTest",
      "iteration": "",
      "fork": "",
      "metric": "executeAction",
      "isWarmup": false,
      "total": {
        "phase": "loadTest",
        "metric": "executeAction",
        "start": 1764707269883,
        "end": 1764707299884,
        "summary": {
          "startTime": 1764707269883,
          "endTime": 1764707299884,
          "minResponseTime": 358400,
          "meanResponseTime": 544238,
          "stdDevResponseTime": 46317,
          "maxResponseTime": 1449983,
          "percentileResponseTime": {
            "50.0": 536575,
            "90.0": 602111,
            "99.0": 684031,
            "99.9": 847871,
            "99.99": 1228799
          },
          "requestCount": 109923,
          "responseCount": 109923,
          "invalid": 0,
          "connectionErrors": 0,
          "requestTimeouts": 0,
          "internalErrors": 0,
          "blockedTime": 0,

// Lots and lots of other info after this

which, when you do the math, only comes out to a throughput of 3545.79 req/s (109923 requests / 30.001 seconds).

Something doesn't seem right to me? Is my hyperfoil script correct?

[hyperfoil@in-vm]$ run app-load-test
Started run 0005
Run 0005, benchmark app-load-test
Agents: in-vm[READY]
Started: 2025/12/02 16:16:24.172    
NAME      STATUS   STARTED       REMAINING  COMPLETED  TOTAL DURATION  DESCRIPTION
loadTest  RUNNING  16:18:24.174   20531 ms                             2 users always

———————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————
Total stats from run 0005
PHASE     METRIC         THROUGHPUT     REQUESTS  MEAN       STD_DEV    MAX        p50        p90        p99        p99.9    p99.99   TIMEOUTS  ERRORS  BLOCKED  2xx     3xx  4xx  5xx  CACHE
warmup    executeAction  3608.88 req/s    433073  552.57 μs  459.24 μs  209.72 ms  540.67 μs  618.50 μs  786.43 μs  1.50 ms  3.26 ms         0       0     0 ns  433073    0    0    0      0
loadTest  executeAction  3650.81 req/s    109528  546.22 μs   58.81 μs    1.96 ms  536.58 μs  610.30 μs  733.18 μs  1.09 ms  1.43 ms         0       0     0 ns  109528    0    0    0      0
[hyperfoil@in-vm]$ wrk -t2 -c100 -d30s http://localhost:8080/fruits
Running 30s test @ http://localhost:8080/fruits
  2 threads and 100 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    10.61ms    6.41ms 100.14ms   85.04%
    Req/Sec   9119.10   1694.06  10346.00     96.77
  282692 requests in 30.012s,   1.48GB read
Requests/sec: 9419.30
Transfer/sec:  50.40MB

[franz1981]

I am on PTO, atm. But looking at this I bet the problem are that the Hyperfoil scripts have just 2 users, making 98 connections to not be used. Wrk instead is both using 100 connections used by 100 users, and that's why it flies. You should use 100 users instead.

One note: you probably don't need 100 users/connections but 10 should be enough since you achieved 1/3 of your 100 users/connections TPS using just 2.

[edeandrea]

Go enjoy PTO!

[edeandrea]

FYI when I bump to 100 users I get 8372.63 req/s

[edeandrea]

And with 50 users (on my local machine with no cpu isolation whatsoever):

[hyperfoil@in-vm]$ run app-load-test
Started run 0010
Total stats from run 0010
PHASE     METRIC         THROUGHPUT     REQUESTS  MEAN     STD_DEV  MAX       p50      p90      p99       p99.9     p99.99    TIMEOUTS  ERRORS  BLOCKED  2xx      3xx  4xx  5xx  CACHE
warmup    executeAction  8862.38 req/s   1063618  5.64 ms  2.84 ms  55.05 ms  4.95 ms  8.98 ms  16.12 ms  27.66 ms  39.32 ms         0       0     0 ns  1063618    0    0    0      0
loadTest  executeAction  8728.55 req/s    261900  5.73 ms  2.87 ms  55.57 ms  5.01 ms  9.18 ms  16.45 ms  26.87 ms  37.75 ms         0       0     0 ns   261900    0    0    0      0

[hyperfoil@in-vm]$ wrk -t2 -c100 -d30s --latency http://localhost:8080/fruits
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    11.29ms    6.96ms  95.42ms   86.08%
    Req/Sec   8571.03   1591.74  9913.00     96.77
  Latency Distribution
     50.000%      9.70ms
     75.000%     14.09ms
     90.000%     19.79ms
     99.000%     37.49ms
     99.900%     56.10ms
     99.990%     74.45ms
     99.999%     90.18ms
    100.000%     95.42ms

  Detailed Percentile Spectrum
       Value     Percentile TotalCount 1/(1-Percentile)

        1.56 0.000000000000          1           1.00
        5.93 0.100000000000      27617           1.11
        6.16 0.200000000000      57074           1.25
        6.36 0.300000000000      80891           1.43
        7.41 0.400000000000     106401           1.67
        9.70 0.500000000000     133808           2.00
       10.09 0.550000000000     148658           2.22
       10.35 0.600000000000     162204           2.50
       10.68 0.650000000000     173927           2.86
       12.98 0.700000000000     186075           3.33
       14.09 0.750000000000     199524           4.00
       14.35 0.775000000000     205909           4.44
       14.68 0.800000000000     212786           5.00
       15.53 0.825000000000     219319           5.71
       17.96 0.850000000000     226426           6.67
       18.61 0.875000000000     233039           8.00
       19.01 0.887500000000     236250           8.89
       19.79 0.900000000000     239330          10.00
       21.50 0.912500000000     242401          11.43
       22.41 0.925000000000     245846          13.33
       23.20 0.937500000000     249091          16.00
       24.12 0.943750000000     250671          17.78
       25.43 0.950000000000     252376          20.00
       26.35 0.956250000000     254077          22.86
       27.13 0.962500000000     255871          26.67
       28.70 0.968750000000     257361          32.00
       29.75 0.971875000000     258181          35.56
       30.54 0.975000000000     259136          40.00
       31.06 0.978125000000     259798          45.71
       32.24 0.981250000000     260641          53.33
       34.34 0.984375000000     261586          64.00
       34.87 0.985937500000     261945          71.11
       35.65 0.987500000000     262412          80.00
       36.70 0.989062500000     262770          91.43
       38.01 0.990625000000     263159         106.67
       39.32 0.992187500000     263620         128.00
       39.85 0.992968750000     263747         142.22
       41.16 0.993750000000     263978         160.00
       42.21 0.994531250000     264167         182.86
       43.25 0.995312500000     264394         213.33
       44.56 0.996093750000     264588         256.00
       45.61 0.996484375000     264693         284.44
       46.66 0.996875000000     264780         320.00
       47.71 0.997265625000     264903         365.71
       49.28 0.997656250000     264997         426.67
       50.86 0.998046875000     265097         512.00
       51.64 0.998242187500     265156         568.89
       52.43 0.998437500000     265192         640.00
       53.74 0.998632812500     265249         731.43
       55.05 0.998828125000     265304         853.33
       56.10 0.999023437500     265349        1024.00
       57.15 0.999121093750     265377        1137.78
       58.20 0.999218750000     265399        1280.00
       59.77 0.999316406250     265427        1462.86
       60.82 0.999414062500     265451        1706.67
       62.13 0.999511718750     265476        2048.00
       63.18 0.999560546875     265489        2275.56
       64.23 0.999609375000     265502        2560.00
       65.27 0.999658203125     265522        2925.71
       66.06 0.999707031250     265528        3413.33
       68.68 0.999755859375     265542        4096.00
       69.21 0.999780273438     265550        4551.11
       69.73 0.999804687500     265557        5120.00
       70.25 0.999829101563     265560        5851.43
       71.83 0.999853515625     265568        6826.67
       73.40 0.999877929688     265573        8192.00
       73.92 0.999890136719     265576        9102.22
       74.97 0.999902343750     265582       10240.00
       75.50 0.999914550781     265583       11702.86
       77.59 0.999926757813     265586       13653.33
       78.64 0.999938964844     265591       16384.00
       78.64 0.999945068359     265591       18204.44
       80.74 0.999951171875     265594       20480.00
       80.74 0.999957275391     265594       23405.71
       84.41 0.999963378906     265596       27306.67
       84.93 0.999969482422     265597       32768.00
       85.46 0.999972534180     265599       36408.89
       85.46 0.999975585938     265599       40960.00
       86.51 0.999978637695     265600       46811.43
       88.08 0.999981689453     265601       54613.33
       88.08 0.999984741211     265601       65536.00
       89.65 0.999986267090     265602       72817.78
       89.65 0.999987792969     265602       81920.00
       90.18 0.999989318848     265603       93622.86
       90.18 0.999990844727     265603      109226.67
       90.18 0.999992370605     265603      131072.00
       91.75 0.999993133545     265604      145635.56
       91.75 0.999993896484     265604      163840.00
       91.75 0.999994659424     265604      187245.71
       91.75 0.999995422363     265604      218453.33
       91.75 0.999996185303     265604      262144.00
       95.42 0.999996566772     265605      291271.11
       95.42 1.000000000000     265605
#[Mean    =        11.29, StdDeviation   =         6.96]
#[Max     =        95.42, Total count    =       265605]
#[Buckets =           29, SubBuckets     =          256]
----------------------------------------------------------
  265702 requests in 30.007s,   1.39GB read
Requests/sec: 8854.67
Transfer/sec:  47.38MB

[franz1981]

Yeah consider that constant rate in hyperfoil is still not 1:1 with wrk and for small runs you should set "variance" to false as well - which is what wrk does, ootb.

Now re

Something doesn't seem right to me? Is my hyperfoil script correct?

I hope, by setting the proper number of users, results will more in line with your expectations

[edeandrea]

Yeah consider that constant rate in hyperfoil is still not 1:1 with wrk and for small runs you should set "variance" to false as well - which is what wrk does, ootb.

Is always the correct type of phase? According to https://hyperfoil.io/docs/user-guide/benchmark/phases/, the variance attribute is only available in constantRate and increasingRate.

[franz1981]

My bad, you are right 👍
Yep, forget about the variance attribute, you're right.

Re wrk, when you set c100, the same Hyperfoil benchmark using "always" should have both connections and users to be set to 100, but as you can see, probably 100 is too generous based on the RTT with the DBMS (which sadly depends by how fast is the machine running the DBMS) and you can get similar peak throughput with lower values for both

[franz1981]

I would suggest to use Hyperfoil scripts to get the ramp up under control; e.g. if the application is warming up and receive a too big load, it would cause its worker pool to grow larger than it would be at steady state, while hot.
In our case we have fixed concurrency (number of connections) but still, if the RTT to serve one connection is small enough, the worker pool won't need to grow till hitting Mac concurrency.
Clearly this is more a concern with a blocking worker pool, whilst with Virtual Thread, not as much