NO_PROXY variable not honored

[waldner]

Running fluentd-kubernetes-daemonset:v1.14.6-debian-elasticsearch7-1.0 inside a k8s cluster behind a proxy. The container has the HTTP_PROXY/HTTPS_PROXY (and their lowercase version) variables set, which are honored; however, the proxy should not be used to connect to the k8s API, so I set NO_PROXY (and no_proxy) to kubernetes,10.43.0.1,kubernetes.default.svc, yet it looks like the API is not being accessed directly. Here are some errors from the log:

Successfully installed fluent-plugin-kubernetes-objects-1.1.12
1 gem installed
2022-07-20 13:13:02 +0000 [info]: parsing config file is succeeded path="/fluentd/etc/fluent.conf"
2022-07-20 13:13:02 +0000 [info]: gem 'fluent-plugin-concat' version '2.5.0'
2022-07-20 13:13:02 +0000 [info]: gem 'fluent-plugin-dedot_filter' version '1.0.0'
2022-07-20 13:13:02 +0000 [info]: gem 'fluent-plugin-detect-exceptions' version '0.0.14'
2022-07-20 13:13:02 +0000 [info]: gem 'fluent-plugin-elasticsearch' version '5.1.5'
2022-07-20 13:13:02 +0000 [info]: gem 'fluent-plugin-grok-parser' version '2.6.2'
2022-07-20 13:13:02 +0000 [info]: gem 'fluent-plugin-json-in-json-2' version '1.0.2'
2022-07-20 13:13:02 +0000 [info]: gem 'fluent-plugin-kubernetes-objects' version '1.1.12'
2022-07-20 13:13:02 +0000 [info]: gem 'fluent-plugin-kubernetes_metadata_filter' version '2.9.5'
2022-07-20 13:13:02 +0000 [info]: gem 'fluent-plugin-multi-format-parser' version '1.0.0'
2022-07-20 13:13:02 +0000 [info]: gem 'fluent-plugin-parser-cri' version '0.1.1'
2022-07-20 13:13:02 +0000 [info]: gem 'fluent-plugin-prometheus' version '2.0.2'
2022-07-20 13:13:02 +0000 [info]: gem 'fluent-plugin-record-modifier' version '2.1.0'
2022-07-20 13:13:02 +0000 [info]: gem 'fluent-plugin-rewrite-tag-filter' version '2.4.0'
2022-07-20 13:13:02 +0000 [info]: gem 'fluent-plugin-systemd' version '1.0.5'
2022-07-20 13:13:02 +0000 [info]: gem 'fluentd' version '1.14.6'
2022-07-20 13:13:02 +0000 [warn]: [filter_kube_metadata] !! The environment variable 'K8S_NODE_NAME' is not set to the node name which can affect the API server and watch efficiency !!
#<Thread:0x00007fe581a3da90 run> terminated with exception (report_on_exception is true):
/fluentd/vendor/bundle/ruby/2.7.0/gems/fluent-plugin-kubernetes_metadata_filter-2.9.5/lib/fluent/plugin/kubernetes_metadata_watch_pods.rb:87:in `rescue in start_pod_watch': start_pod_watch: Exception encountered setting up pod watch from Kubernetes API v1 endpoint https://10.43.0.1:443/api: pods is forbidden: User "system:serviceaccount:myns:default" cannot list resource "pods" in API group "" at the cluster scope ({"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"pods is forbidden: User \\"system:serviceaccount:myns:default\\" cannot list resource \\"pods\\" in API group \\"\\" at the cluster scope","reason":"Forbidden","details":{"kind":"pods"},"code":403} (Fluent::ConfigError)
)
	from /fluentd/vendor/bundle/ruby/2.7.0/gems/fluent-plugin-kubernetes_metadata_filter-2.9.5/lib/fluent/plugin/kubernetes_metadata_watch_pods.rb:78:in `start_pod_watch'
	from /fluentd/vendor/bundle/ruby/2.7.0/gems/fluent-plugin-kubernetes_metadata_filter-2.9.5/lib/fluent/plugin/kubernetes_metadata_watch_pods.rb:32:in `set_up_pod_thread'
/fluentd/vendor/bundle/ruby/2.7.0/gems/kubeclient-4.9.3/lib/kubeclient/common.rb:130:in `rescue in handle_exception': pods is forbidden: User "system:serviceaccount:myns:default" cannot list resource "pods" in API group "" at the cluster scope (Kubeclient::HttpError)
	from /fluentd/vendor/bundle/ruby/2.7.0/gems/kubeclient-4.9.3/lib/kubeclient/common.rb:120:in `handle_exception'
	from /fluentd/vendor/bundle/ruby/2.7.0/gems/kubeclient-4.9.3/lib/kubeclient/common.rb:350:in `get_entities'
	from /fluentd/vendor/bundle/ruby/2.7.0/gems/kubeclient-4.9.3/lib/kubeclient/common.rb:224:in `block (2 levels) in define_entity_methods'
	from /fluentd/vendor/bundle/ruby/2.7.0/gems/kubeclient-4.9.3/lib/kubeclient/common.rb:101:in `method_missing'
	from /fluentd/vendor/bundle/ruby/2.7.0/gems/fluent-plugin-kubernetes_metadata_filter-2.9.5/lib/fluent/plugin/kubernetes_metadata_watch_pods.rb:102:in `get_pods_and_start_watcher'
	from /fluentd/vendor/bundle/ruby/2.7.0/gems/fluent-plugin-kubernetes_metadata_filter-2.9.5/lib/fluent/plugin/kubernetes_metadata_watch_pods.rb:79:in `start_pod_watch'
	from /fluentd/vendor/bundle/ruby/2.7.0/gems/fluent-plugin-kubernetes_metadata_filter-2.9.5/lib/fluent/plugin/kubernetes_metadata_watch_pods.rb:32:in `set_up_pod_thread'
/fluentd/vendor/bundle/ruby/2.7.0/gems/rest-client-2.1.0/lib/restclient/abstract_response.rb:249:in `exception_with_response': 403 Forbidden (RestClient::Forbidden)
	from /fluentd/vendor/bundle/ruby/2.7.0/gems/rest-client-2.1.0/lib/restclient/abstract_response.rb:129:in `return!'
	from /fluentd/vendor/bundle/ruby/2.7.0/gems/rest-client-2.1.0/lib/restclient/request.rb:836:in `process_result'
	from /fluentd/vendor/bundle/ruby/2.7.0/gems/rest-client-2.1.0/lib/restclient/request.rb:743:in `block in transmit'
	from /usr/local/lib/ruby/2.7.0/net/http.rb:933:in `start'
	from /fluentd/vendor/bundle/ruby/2.7.0/gems/rest-client-2.1.0/lib/restclient/request.rb:727:in `transmit'
	from /fluentd/vendor/bundle/ruby/2.7.0/gems/rest-client-2.1.0/lib/restclient/request.rb:163:in `execute'
	from /fluentd/vendor/bundle/ruby/2.7.0/gems/rest-client-2.1.0/lib/restclient/request.rb:63:in `execute'
	from /fluentd/vendor/bundle/ruby/2.7.0/gems/rest-client-2.1.0/lib/restclient/resource.rb:51:in `get'
	from /fluentd/vendor/bundle/ruby/2.7.0/gems/kubeclient-4.9.3/lib/kubeclient/common.rb:352:in `block in get_entities'
	from /fluentd/vendor/bundle/ruby/2.7.0/gems/kubeclient-4.9.3/lib/kubeclient/common.rb:121:in `handle_exception'
	from /fluentd/vendor/bundle/ruby/2.7.0/gems/kubeclient-4.9.3/lib/kubeclient/common.rb:350:in `get_entities'
	from /fluentd/vendor/bundle/ruby/2.7.0/gems/kubeclient-4.9.3/lib/kubeclient/common.rb:224:in `block (2 levels) in define_entity_methods'
	from /fluentd/vendor/bundle/ruby/2.7.0/gems/kubeclient-4.9.3/lib/kubeclient/common.rb:101:in `method_missing'
	from /fluentd/vendor/bundle/ruby/2.7.0/gems/fluent-plugin-kubernetes_metadata_filter-2.9.5/lib/fluent/plugin/kubernetes_metadata_watch_pods.rb:102:in `get_pods_and_start_watcher'
	from /fluentd/vendor/bundle/ruby/2.7.0/gems/fluent-plugin-kubernetes_metadata_filter-2.9.5/lib/fluent/plugin/kubernetes_metadata_watch_pods.rb:79:in `start_pod_watch'
	from /fluentd/vendor/bundle/ruby/2.7.0/gems/fluent-plugin-kubernetes_metadata_filter-2.9.5/lib/fluent/plugin/kubernetes_metadata_watch_pods.rb:32:in `set_up_pod_thread'

On another cluster running without the proxy (everything else the same), no error is produced.

[github-actions]

This issue has been automatically marked as stale because it has been open 90 days with no activity. Remove stale label or comment or this issue will be closed in 30 days

[waldner]

The issue is still relevant.

[github-actions]

This issue has been automatically marked as stale because it has been open 90 days with no activity. Remove stale label or comment or this issue will be closed in 30 days

[waldner]

The issue is still relevant.

[github-actions]

This issue has been automatically marked as stale because it has been open 90 days with no activity. Remove stale label or comment or this issue will be closed in 30 days

[waldner]

The issue is still relevant.

[mikaelkrief]

any solution to this issue ?

[vittico]

Is there any solution available?

[github-actions]

This issue has been automatically marked as stale because it has been open 90 days with no activity. Remove stale label or comment or this issue will be closed in 30 days