What about high latency of wrk access interface on ubuntu?

[ponponon]

I have an api service http://192.168.38.223:9100/

Deployed using docker on machine 192.168.38.223

╰─➤  docker ps -a | grep proxy         
12b8b5801e73   ponponon/proxy-king:2023.09.21.1           "python api.py"          38 minutes ago   Up 38 minutes         0.0.0.0:9100->9100/tcp, :::9100->9100/tcp                                                                                                                                                      proxy-king

I then use wrk on mac to stress test the http://192.168.38.223:9100/ interface

The wrk is installed by brew install wrk.

╰─➤  wrk --version                                  
wrk 4.2.0 [kqueue] Copyright (C) 2012 Will Glozer
Usage: wrk <options> <url>                            
Options:                                            
 -c, --connections <N>  Connections to keep open   
 -d, --duration    <T>  Duration of test           
 -t, --threads     <N>  Number of threads to use   

 -s, --script      <S>  Load Lua script file       
 -H, --header      <H>  Add header to request      
     --latency          Print latency statistics   
     --timeout     <T>  Socket/request timeout     
 -v, --version          Print version details      

Numeric arguments may include a SI unit (1k, 1M, 1G)
Time arguments may include a time unit (2s, 2m, 2h)

╰─➤  wrk http://192.168.38.223:9100/ -t32 -c64 -d 30                                                                                                                     130 ↵
Running 30s test @ http://192.168.38.223:9100/
  32 threads and 64 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     3.51ms    2.99ms  64.81ms   91.05%
    Req/Sec   644.17    204.94     1.16k    66.50%
  615969 requests in 30.04s, 92.72MB read
Requests/sec:  20503.44
Transfer/sec:      3.09MB

The result is very good, the RPS can reach 2w+.

I then used the wrk tool on 192.168.38.223 to stress test the http://192.168.38.223:9100/ interface again

192.168.38.223 The machine is using ubuntu 22.04.

╰─➤  uname -a
Linux T4GPU 5.15.0-84-generic #93-Ubuntu SMP Tue Sep 5 17:16:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

The installed wrk is apt installed apt install wrk.

╰─➤  wrk --version                                                                                                                                                         1 ↵
wrk debian/4.1.0-3build1 [epoll] Copyright (C) 2012 Will Glozer
Usage: wrk <options> <url>                            
Options:                                            
 -c, --connections <N>  Connections to keep open   
 -d, --duration    <T>  Duration of test           
 -t, --threads     <N>  Number of threads to use   

 -s, --script      <S>  Load Lua script file       
 -H, --header      <H>  Add header to request      
     --latency          Print latency statistics   
     --timeout     <T>  Socket/request timeout     
 -v, --version          Print version details      

Numeric arguments may include a SI unit (1k, 1M, 1G)
Time arguments may include a time unit (2s, 2m, 2h)

But something unexpected happened, the RPS became very low, only 1.4k, and the latency became very high, 45ms! This is unusual!

╰─➤  ./wrk http://192.168.38.223:9100/ -t32 -c64 -d 30
Running 30s test @ http://192.168.38.223:9100/
  32 threads and 64 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    44.67ms    2.24ms  52.34ms   81.85%
    Req/Sec    44.78      6.01    60.00     94.41%
  43079 requests in 30.09s, 6.49MB read
Requests/sec:   1431.51
Transfer/sec:    220.89KB

I ran a python script on the 192.168.38.223 machine

import time
import requests


s=time.time()
response=requests.get('http://192.168.38.223:9100/')
e=time.time()

print(f'{round((e-s)*1000,2)} ms')

output result

4.34 ms

As you can see, there are no latency issues with accessing using python scripts, but why is the latency of wrk access so high? What are the ways to troubleshoot this?

[qsantos]

You might be observing #488. Add -H 'Connection: close' to your wrk invocation.