ETCD connection in non tls mode in istio-proxy service mesh

[aru88]

Hello,
we have a kubernates cluster and istio-proxy service mesh.
we have deployed etcd in no tls mode .
The variables such as TLS_ENABLED, PEER_AUTO_TLS_ENABLED, and PEER_CERT_AUTH_ENABLED are set to false, indicating that TLS and client authentication are disabled.

in same service mesh we have deployed a python service which is connecting with etcd using etcd3 client.
both etcd and pyhton service has istio-proxy as sidecar. it's the istio-proxy sidecar which is handling mtls communication between both.

but while trying to read etcd i am getting this error

{
"detail": "<_InactiveRpcError of RPC that terminated with:\n\tstatus = StatusCode.INVALID_ARGUMENT\n\tdetails = "etcdserver: user name is empty"\n\tdebug_error_string = "UNKNOWN:Error received from peer {grpc_message:"etcdserver: user name is empty", grpc_status:3, created_time:"2023-12-01T19:00:31.101268091+00:00"}"\n> An error has occurred. "
}

so if TLS and client authentication are disabled. why etcd is expecting to pass username and password.

this is the code snippet i am using
con = etcd3.client(
host=CONFIG.get(CONFIG_ETCD_HOST),
port=CONFIG.get(CONFIG_ETCD_PORT)
)
con.get(key)

Please help.

Thanks

[jmhbnz]

Hey @aru88 - Thanks for your question. Is basic username and password auth enabled for your etcd instance? Refer to: https://etcd.io/docs/v3.5/op-guide/authentication/rbac/#enabling-authentication

Please provide your full etcd configuration file or parameters (redacted if neccessary) so we can answer more concretely.

[aru88]

Here is the etcd pod configs :
Name: data-distributed-coordinator-ed-0
Namespace: poc-ns
Priority: 0
Service Account: data-distributed-coordinator-ed-service-account
Node: k8swrk-lab-01/192.128.255.3
Start Time: Wed, 29 Nov 2023 14:55:42 +0530
Labels: app=data-distributed-coordinator-ed
app.kubernetes.io/instance=data-distributed-coordinator-ed
app.kubernetes.io/managed-by=Helm
app.kubernetes.io/name=data-distributed-coordinator-ed
app.kubernetes.io/version=data-distributed-coordinator-ed-9.1.0-16
controller-revision-hash=data-distributed-coordinator-ed-59b4f7fc6f
mesh-controller-sm-iam-poc-v1-1-access=true
security.istio.io/tlsMode=istio
service.istio.io/canonical-name=data-distributed-coordinator-ed
service.istio.io/canonical-revision=data-distributed-coordinator-ed-9.1.0-16
statefulset.kubernetes.io/pod-name=data-distributed-coordinator-ed-0
Annotations: cni.projectcalico.org/containerID: e9d217594575ad15e2b9c4e8330a849f75cbd587aac48cf3dc2d7fed98b5b6e7
cni.projectcalico.org/podIP: 10.190.32.199/32
cni.projectcalico.org/podIPs: 10.190.32.199/32
com/product-name: Distributed Coordinator ED
com/product-number: PPPPPPP
com/product-revision: 9.1.0
kubectl.kubernetes.io/default-container: dced
kubectl.kubernetes.io/default-logs-container: dced
kubernetes.io/psp: plc-76174078fedb5f5a19779c384bf4be
prometheus.io/path: /stats/prometheus
prometheus.io/port: 15020
prometheus.io/scrape: true
sidecar.istio.io/status:
{"initContainers":["istio-init"],"containers":["istio-proxy"],"volumes":["workload-socket","credential-socket","workload-certs","istio-env...
Status: Running
IP: 10.140.52.179
IPs:
IP: 10.140.52.179
Controlled By: StatefulSet/data-distributed-coordinator-ed
Init Containers:
init:
Container ID: containerd://3e76a9479f6690c5f30c4508abd0503c25f3c430b868f36f715740cb5836cc66
Image: prmdocker.rnd.se/proj-adp-data-distributed-coordinator-ed-drop/data-distributed-coordinator-ed:9.1.0-16
Image ID: prmdocker.rnd.se/proj-adp-data-distributed-coordinator-ed-drop/data-distributed-coordinator-ed@sha256:8adef686a250702db7879ca84c8d5e376bf5d5dde85dcd2aa1da90a21e9751bb
Port:
Host Port:
Command:
/bin/bash
Args:
-c
/opt/redirect/stdout-redirect -config=/etc/stdout-redirect-config.yaml -format=json -redirect=stdout -container=init -service-id=data-distributed-coordinator-ed -run="/usr/local/bin/scripts/addMemberBootstrapEnabled.sh"
State: Terminated
Reason: Completed
Exit Code: 0
Started: Wed, 29 Nov 2023 14:56:23 +0530
Finished: Wed, 29 Nov 2023 14:56:24 +0530
Ready: True
Restart Count: 0
Limits:
cpu: 500m
memory: 500Mi
Requests:
cpu: 200m
memory: 200Mi
Environment:
ETCDCTL_API: 3
ETCDCTL_ENDPOINTS: data-distributed-coordinator-ed.poc-ns:2379
ENTRYPOINT_PIPE_TIMEOUT: 5
POD_NAME: data-distributed-coordinator-ed-0 (v1:metadata.name)
CONTAINER_NAME: init
NAMESPACE: poc-ns (v1:metadata.namespace)
FIFO_DIR: /fifo
TZ: UTC
DCED_PORT: 2379
TLS_ENABLED: false
BOOTSTRAP_ENABLED: true
TRUSTED_CA: /data/combinedca/cacertbundle.pem
ETCD_NAME: data-distributed-coordinator-ed-0 (v1:metadata.name)
ETCD_INITIAL_ADVERTISE_PEER_URLS: http://$(ETCD_NAME).data-distributed-coordinator-ed-peer.poc-ns.svc.cluster.local:2380
ETCD_DATA_DIR: /data
ACL_ROOT_PASSWORD: <set to the key 'etcdpasswd' in secret 'data-distributed-coordinator-creds'> Optional: false
Mounts:
/data from data (rw)
/fifo from fifo (rw)
istio-init:
Container ID: containerd://f228517dc10533552968e5a8ec052b521ebc02ed9f4145cc22c90181926e6890
Image: prmdocker.rnd.se/proj-adp-gs-service-mesh/mesh-proxy:12.1.0-6
Image ID: prmdocker.rnd.se/proj-adp-gs-service-mesh/mesh-proxy@sha256:0cebaa2a047c68981349919677a685d6f04320e2970728ecd525b289aea52b8b
Port:
Host Port:
Args:
istio-iptables
-p
15001
-z
15006
-u
201507
-m
REDIRECT
-i
*
-x

  -b
  *
  -d
  15090,15021,15020
  --log_output_level=default:info
State:          Terminated
  Reason:       Completed
  Exit Code:    0
  Started:      Wed, 29 Nov 2023 14:56:25 +0530
  Finished:     Wed, 29 Nov 2023 14:56:25 +0530
Ready:          True
Restart Count:  0
Limits:
  cpu:     2
  memory:  1Gi
Requests:
  cpu:     10m
  memory:  10Mi
Environment:
  INSTANCE_IPS:      (v1:status.podIPs)
  LOG_FILE_SUFFIX:  _init
  LOG_OUTPUT:       stdout
  TZ:               UTC
  POD_NAME:         data-distributed-coordinator-ed-0 (v1:metadata.name)
  POD_NAMESPACE:   poc-ns (v1:metadata.namespace)
  CONTAINER_NAME:   istio-init
  SERVICE_ID:       mesh-proxy
  LOG_SCHEMA:       none
Mounts:             <none>

Containers:
istio-proxy:
Container ID: containerd://721a3eb5250408947c1b1fd1d0bf01f274351b1f878a765868722cfa74c2fc6f
Image: prmdocker.rnd.se/proj-adp-gs-service-mesh/mesh-proxy:12.1.0-6
Image ID: prmdocker.rnd.se/proj-adp-gs-service-mesh/mesh-proxy@sha256:0cebaa2a047c68981349919677a685d6f04320e2970728ecd525b289aea52b8b
Ports: 80/TCP, 15001/TCP, 15006/TCP, 15020/TCP, 15021/TCP, 15090/TCP
Host Ports: 0/TCP, 0/TCP, 0/TCP, 0/TCP, 0/TCP, 0/TCP
Args:
proxy
sidecar
--domain
$(POD_NAMESPACE).svc.cluster.local
--proxyLogLevel=info
--proxyComponentLogLevel=misc:error
--log_output_level=default:info
--concurrency
2
State: Running
Started: Wed, 29 Nov 2023 14:56:26 +0530
Ready: True
Restart Count: 0
Limits:
cpu: 2
memory: 1Gi
Requests:
cpu: 100m
memory: 128Mi
Liveness: http-get http://:15021/healthz/ready delay=0s timeout=1s period=2s #success=1 #failure=60
Readiness: http-get http://:15021/healthz/ready delay=0s timeout=1s period=2s #success=1 #failure=30
Startup: http-get http://:15021/healthz/ready delay=0s timeout=5s period=1s #success=1 #failure=300
Environment:
REWRITE_PROBE_LEGACY_LOCALHOST_DESTINATION: true
JWT_POLICY: third-party-jwt
PILOT_CERT_PROVIDER: istiod
CA_ADDR: cert-manager-istio-csr-iam-poc.iam-sm-cert.svc:443
WORKLOAD_RSA_KEY_SIZE: 3072
POD_NAME: data-distributed-coordinator-ed-0 (v1:metadata.name)
POD_NAMESPACE: poc-ns (v1:metadata.namespace)
CONTAINER_NAME: istio-proxy
SERVICE_ID: mesh-proxy
INSTANCE_IP: (v1:status.podIP)
INSTANCE_IPS: (v1:status.podIPs)
SERVICE_ACCOUNT: (v1:spec.serviceAccountName)
HOST_IP: (v1:status.hostIP)
PROXY_CONFIG: {"discoveryAddress":"istiod.iam-sm-poc.svc:15012","tracing":{"zipkin":{"address":"localhost:9411"}},"holdApplicationUntilProxyStarts":true}

  ISTIO_META_POD_PORTS:                        [
                                                   {"name":"peer","containerPort":2380,"protocol":"TCP"}
                                                   ,{"name":"client","containerPort":2379,"protocol":"TCP"}
                                               ]
  ISTIO_META_APP_CONTAINERS:                   dced
  ISTIO_META_CLUSTER_ID:                       Kubernetes
  ISTIO_META_NODE_NAME:                         (v1:spec.nodeName)
  ISTIO_META_INTERCEPTION_MODE:                REDIRECT
  ISTIO_META_WORKLOAD_NAME:                    data-distributed-coordinator-ed
  ISTIO_META_OWNER:                            kubernetes://apis/apps/v1/namespaces/poc-ns/statefulsets/data-distributed-coordinator-ed
  ISTIO_META_MESH_ID:                          cluster.local
  TRUST_DOMAIN:                                cluster.local
  LOG_OUTPUT:                                  stdout
  LOG_SCHEMA:                                  none
  TZ:                                          UTC
  ISTIO_KUBE_APP_PROBERS:                      {"/app-health/dced/livez":{"httpGet":{"path":"/health/liveness","port":9000,"scheme":"HTTP"},"timeoutSeconds":15},"/app-health/dced/readyz":{"httpGet":{"path":"/health/readiness","port":9000,"scheme":"HTTP"},"timeoutSeconds":15},"/app-health/dced/startupz":{"httpGet":{"path":"/health/startup","port":9000,"scheme":"HTTP"},"timeoutSeconds":15}}
Mounts:
  /etc/istio/pod from istio-podinfo (rw)
  /etc/istio/proxy from istio-envoy (rw)
  /var/lib/istio/data from istio-data (rw)
  /var/run/secrets/credential-uds from credential-socket (rw)
  /var/run/secrets/istio from istiod-ca-cert (rw)
  /var/run/secrets/tokens from istio-token (rw)
  /var/run/secrets/workload-spiffe-credentials from workload-certs (rw)
  /var/run/secrets/workload-spiffe-uds from workload-socket (rw)

dced:
Container ID: containerd://3544b8a95e995da9cdfdb8d40589984fbafd778314cf03590a8264ee22c69319
Image: prmdocker.rnd.se/proj-adp-data-distributed-coordinator-ed-drop/data-distributed-coordinator-ed:9.1.0-16
Image ID: prmdocker.rnd.se/proj-adp-data-distributed-coordinator-ed-drop/data-distributed-coordinator-ed@sha256:8adef686a250702db7879ca84c8d5e376bf5d5dde85dcd2aa1da90a21e9751bb
Ports: 2380/TCP, 2379/TCP
Host Ports: 0/TCP, 0/TCP
Command:
/usr/bin/catatonit
Args:
--
/bin/bash
-c
/opt/redirect/stdout-redirect -config=/etc/stdout-redirect-config.yaml -format=json -redirect=stdout -container=dced -service-id=data-distributed-coordinator-ed -run="/usr/local/bin/scripts/entrypoint.sh"
State: Running
Started: Wed, 29 Nov 2023 14:56:30 +0530
Ready: True
Restart Count: 0
Limits:
cpu: 1
memory: 5Gi
Requests:
cpu: 400m
memory: 400Mi
Liveness: http-get http://:15020/app-health/dced/livez delay=15s timeout=15s period=10s #success=1 #failure=3
Readiness: http-get http://:15020/app-health/dced/readyz delay=15s timeout=15s period=10s #success=1 #failure=3
Startup: http-get http://:15020/app-health/dced/startupz delay=5s timeout=15s period=10s #success=1 #failure=12
Environment:
ETCDCTL_API: 3
ETCDCTL_ENDPOINTS: data-distributed-coordinator-ed.poc-ns:2379
POD_NAME: data-distributed-coordinator-ed-0 (v1:metadata.name)
CONTAINER_NAME: dced
NAMESPACE: poc-ns (v1:metadata.namespace)
FIFO_DIR: /fifo
HTTP_PROBE_CMD_DIR: /usr/local/bin/health
HTTP_PROBE_SERVICE_NAME: data-distributed-coordinator-ed
HTTP_PROBE_POD_NAME: data-distributed-coordinator-ed-0 (v1:metadata.name)
HTTP_PROBE_NAMESPACE: poc-ns (v1:metadata.namespace)
HTTP_PROBE_CONTAINER_NAME: dced
HTTP_PROBE_STARTUP_CMD_TIMEOUT_SEC: 15
HTTP_PROBE_READINESS_CMD_TIMEOUT_SEC: 15
HTTP_PROBE_LIVENESS_CMD_TIMEOUT_SEC: 15
DCED_PORT: 2379
TZ: UTC
TLS_ENABLED: false
BOOTSTRAP_ENABLED: true
ENTRYPOINT_CHECKSNUMBER: 60
ENTRYPOINT_RESTART_ETCD: true
ENTRYPOINT_PIPE_TIMEOUT: 5
ENTRYPOINT_DCED_PROCESS_INTERVAL: 5
PEER_CLIENT_KEY_FILE: /run/sec/certs/peer/srvprivkey.pem
PEER_AUTO_TLS_ENABLED: false
TRUSTED_CA: /data/combinedca/cacertbundle.pem
PEER_CERT_AUTH_ENABLED: false
ETCD_INITIAL_CLUSTER_TOKEN: data-distributed-coordinator-ed
ETCD_NAME: data-distributed-coordinator-ed-0 (v1:metadata.name)
ETCD_DATA_DIR: /data
LISTEN_PEER_URLS: http://0.0.0.0:2380
ETCD_LISTEN_CLIENT_URLS: http://0.0.0.0:2379
ETCD_ADVERTISE_CLIENT_URLS: http://$(ETCD_NAME).data-distributed-coordinator-ed.poc-ns:2379
ETCD_INITIAL_ADVERTISE_PEER_URLS: http://$(ETCD_NAME).data-distributed-coordinator-ed-peer.poc-ns.svc.cluster.local:2380
ETCD_PEER_AUTO_TLS: false
ETCD_LISTEN_PEER_URLS: http://0.0.0.0:2380
ACL_ROOT_PASSWORD: <set to the key 'etcdpasswd' in secret 'data-distributed-coordinator-creds'> Optional: false
VALID_PARAMETERS: valid
DB_THRESHOLD_PERCENTAGE: 70
DEFRAGMENT_ENABLE: true
DEFRAGMENT_PERIODIC_INTERVAL: 60
DISprm_ALprm_PEER_INTERVAL: 6
ETCD_AUTO_COMPACTION_MODE: revision
ETCD_AUTO_COMPACTION_RETENTION: 100
ETCD_ELECTION_TIMEOUT: 1000
ETCD_HEARTBEAT_INTERVAL: 100
ETCD_LOGGER: zap
ETCD_LOG_LEVEL: debug
ETCD_MAX_SNAPSHOTS: 3
ETCD_MAX_WALS: 3
ETCD_METRICS: basic
ETCD_QUOTA_BACKEND_BYTES: 1423780000
ETCD_SNAPSHOT_COUNT: 5000
GODEBUG: tls13=1
MONITOR_ALprm_INTERVAL: 5
Mounts:
/data from data (rw)
/fifo from fifo (rw)
Conditions:
Type Status
Initialized True
Ready True
ContainersReady True
PodScheduled True
Volumes:
workload-socket:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium:
SizeLimit:
credential-socket:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium:
SizeLimit:
workload-certs:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium:
SizeLimit:
istio-envoy:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium: Memory
SizeLimit:
istio-data:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium:
SizeLimit:
istio-podinfo:
Type: DownwardAPI (a volume populated by information about the pod)
Items:
metadata.labels -> labels
metadata.annotations -> annotations
istio-token:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 43200
istiod-ca-cert:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: istio-ca-root-cert
Optional: false
data:
Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
ClaimName: data-data-distributed-coordinator-ed-0
ReadOnly: false
fifo:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium: Memory
SizeLimit: 1Mi
QoS Class: Burstable
Node-Selectors:
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 60s
node.kubernetes.io/unreachable:NoExecute op=Exists for 60s
Events:
(

i need to connect using only host and port .

even if i am passing credentials in that case i am getting error {
"detail": "<_InactiveRpcError of RPC that terminated with:\n\tstatus = StatusCode.UNAUTHENTICATED\n\tdetails = "Established channel does not have a sufficient security level to transfer call credential."\n\tdebug_error_string = "UNKNOWN:Error received from peer {created_time:"2023-12-04T17:33:03.632887416+00:00", grpc_status:16, grpc_message:"Established channel does not have a sufficient security level to transfer call credential."}"\n> An error has occurred. "
}

so both options not working for me .

[jmhbnz]

Hey @aru88 - As mentioned in the previous discussion you raised (#17012), the error Established channel does not have a sufficient security level to transfer call credential. is likely to be a bug with the python etcd client you are using.

Refer to: kragniz/python-etcd3#1876.

[aru88]

Thank you for you response. is it possible we can connect with etcd using only host and port . both etcd and python client deployed in non tls mode in istio-proxy service mesh .

[jmhbnz]

Does your cluster currently have auth enabled? Refer to https://etcd.io/docs/v3.5/op-guide/authentication/rbac/#enabling-authentication

If basic authentication is currently enabled in your cluster to connect without needing to pass the username and password you would need to connect as an authenticated user and run for example etcdctcl --user user:password auth disable to disable authentication.

[aru88]

ok