Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

how to disable upstream statistics collection? #292

Open
k0st1an opened this issue Jan 21, 2024 · 4 comments · May be fixed by #293
Open

how to disable upstream statistics collection? #292

k0st1an opened this issue Jan 21, 2024 · 4 comments · May be fixed by #293

Comments

@k0st1an
Copy link

k0st1an commented Jan 21, 2024

how to disable upstream statistics collection?

I want to disable statistics collection for upstream servers to reduce CPU load. How can I do this?

image

Current config

load_module modules/ngx_http_vhost_traffic_status_module.so;

http {
  vhost_traffic_status_zone;

  server {
    listen 4444;
    server_name s1.com;
    location /metrics {
      limit_except GET {
        allow 10.99.0.0/16;
        deny  all;
      }
      vhost_traffic_status_display;
      vhost_traffic_status_display_format prometheus;
    }
  }

  server {
    vhost_traffic_status_filter_by_set_key $status $server_name;
    ...
  }
}
u5surf added a commit to u5surf/nginx-module-vts that referenced this issue Jan 23, 2024
@u5surf
Introduce bypass_upstream_stats vozlt#292
7326fe0 
* it is able to disable intentionally statistics collection for upstream servers.
@u5surf u5surf linked a pull request Jan 23, 2024 that will close this issue
Open
@u5surf
Copy link
Collaborator

u5surf commented Jan 23, 2024

@k0st1an Thanks. Unfortunately, we still cannot prepare the flag to disable the upstream statistics intentionally now.

We are fine what you said there are not good situations if we use the upstream directive we has enable its statistics simultaneously in some cases.

So I introduced the new parameter in #293 that can be through the upstream statistics. Can you review and try this parameter?

@k0st1an
Copy link
Author

k0st1an commented Jan 26, 2024 

nginx version: nginx/1.24.0
built by gcc 12.2.0 (Debian 12.2.0-14)
built with OpenSSL 3.0.12 24 Oct 2023 (running with OpenSSL 3.0.11 19 Sep 2023)
TLS SNI support enabled
configure arguments: --prefix=/etc/nginx --sbin-path=/usr/sbin/nginx --modules-path=/usr/lib/nginx/modules --conf-path=/etc/nginx/nginx.conf --error-log-path=/var/log/nginx/error.log --http-log-path=/var/log/nginx/access.log --pid-path=/var/run/nginx.pid --lock-path=/var/run/nginx.lock --http-client-body-temp-path=/var/cache/nginx/client_temp --http-proxy-temp-path=/var/cache/nginx/proxy_temp --http-fastcgi-temp-path=/var/cache/nginx/fastcgi_temp --http-uwsgi-temp-path=/var/cache/nginx/uwsgi_temp --http-scgi-temp-path=/var/cache/nginx/scgi_temp --user=nginx --group=nginx --with-compat --with-file-aio --with-threads --with-http_addition_module --with-http_auth_request_module --with-http_dav_module --with-http_flv_module --with-http_gunzip_module --with-http_gzip_static_module --with-http_mp4_module --with-http_random_index_module --with-http_realip_module --with-http_secure_link_module --with-http_slice_module --with-http_ssl_module --with-http_stub_status_module --with-http_sub_module --with-http_v2_module --with-mail --with-mail_ssl_module --with-stream --with-stream_realip_module --with-stream_ssl_module --with-stream_ssl_preread_module --with-cc-opt='-g -O2 -ffile-prefix-map=/data/builder/debuild/nginx-1.24.0/debian/debuild-base/nginx-1.24.0=. -fstack-protector-strong -Wformat -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -fPIC' --with-ld-opt='-Wl,-z,relro -Wl,-z,now -Wl,--as-needed -pie' --add-module=/nginx-module-vts-issue-292

When two options are active, the statistics disappear


http {
  vhost_traffic_status_zone;
  vhost_traffic_status_bypass_upstream_stats on;  # <--

  server {
    listen 4444;
    server_name s1.com;
    location /metrics {
      limit_except GET {
        allow 10.99.0.0/16;
        deny  all;
      }
      vhost_traffic_status_display;
      vhost_traffic_status_display_format prometheus;
    }
  }

  server {
    vhost_traffic_status_filter_by_set_key $status $server_name;  # <--
    ...
  }
}
Screenshot 2024-01-26 at 1 40 33 PM

When only vhost_traffic_status_bypass_upstream_stats is set to on, there are no upstream statistics. But I don't see any difference in CPU load yet. I don't understand why this is so. Before that, I used https://deb.myguard.nl/nginx-modules/ repository. I will install it from this repository again and check my guess.

@k0st1an
Copy link
Author

k0st1an commented Jan 27, 2024

The problem reappeared with the https://deb.myguard.nl/nginx-modules/ repository. Configuration as in the first post. CPU throttle spike at 70k RPS

  • 30 server (original nginx, without VTS) - 80k
  • 31 server - 70k
Screenshot 2024-01-26 at 6 14 48 PM 
$ dpkg -l | grep vhost
ii  libnginx-mod-http-vhost-traffic-status 1.25.3-4myguard21~bookworm              amd64        Nginx virtual host traffic status module

With upstreams disabled, the difference in CPU load is 2-3%. But I would like to test with the option vhost_traffic_status_status_filter_by_set_key $status $server_name;. But the stats are missing as I wrote above.

@u5surf
Copy link
Collaborator

u5surf commented Jan 31, 2024

@k0st1an
It seems that I was misleaded your truely problem. So actually is your problem which you wanna really solve now the CPU utilization suddenly spiked?
It is necessary for us to solve the problem if it can be reproduced absolutely the same situation for same configuration and the same request per sec. If it does that, we can get some profile with which we can investigate more such the situation.

https://gcc.gnu.org/onlinedocs/gcc-12.1.0/gcc/Instrumentation-Options.html

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Introduce bypass_upstream_stats #292 u5surf/nginx-module-vts
2 participants
@k0st1an @u5surf