K8s: Name or service not known

[mgazzali]

I have installed st2 via k8 HA deployment
I have installed utility packs from stackstorm exchange
Encountered this error few times

{
  "stdout": "",
  "stderr": "st2.actions.python.St2ActionsListAction: DEBUG    Calling client method \"get_all\" with kwargs \"{'limit': 10}\"
Traceback (most recent call last):
  File \"/opt/stackstorm/st2/lib/python3.8/site-packages/urllib3/connection.py\", line 174, in _new_conn
    conn = connection.create_connection(
  File \"/opt/stackstorm/st2/lib/python3.8/site-packages/urllib3/util/connection.py\", line 72, in create_connection
    for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM):
  File \"/usr/lib/python3.8/socket.py\", line 918, in getaddrinfo
    for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
socket.gaierror: [Errno -2] Name or service not known

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File \"/opt/stackstorm/st2/lib/python3.8/site-packages/urllib3/connectionpool.py\", line 703, in urlopen
    httplib_response = self._make_request(
  File \"/opt/stackstorm/st2/lib/python3.8/site-packages/urllib3/connectionpool.py\", line 386, in _make_request
    self._validate_conn(conn)
  File \"/opt/stackstorm/st2/lib/python3.8/site-packages/urllib3/connectionpool.py\", line 1040, in _validate_conn
    conn.connect()
  File \"/opt/stackstorm/st2/lib/python3.8/site-packages/urllib3/connection.py\", line 358, in connect
    self.sock = conn = self._new_conn()
  File \"/opt/stackstorm/st2/lib/python3.8/site-packages/urllib3/connection.py\", line 186, in _new_conn
    raise NewConnectionError(
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPSConnection object at 0x7ffff63b1130>: Failed to establish a new connection: [Errno -2] Name or service not known

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File \"/opt/stackstorm/st2/lib/python3.8/site-packages/requests/adapters.py\", line 439, in send
    resp = conn.urlopen(
  File \"/opt/stackstorm/st2/lib/python3.8/site-packages/urllib3/connectionpool.py\", line 785, in urlopen
    retries = retries.increment(
  File \"/opt/stackstorm/st2/lib/python3.8/site-packages/urllib3/util/retry.py\", line 592, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='st2api.eog-prd-3-7.svc.cluster.local', port=9101): Max retries exceeded with url: /v1/actions?limit=10 (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7ffff63b1130>: Failed to establish a new connection: [Errno -2] Name or service not known'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File \"/opt/stackstorm/st2/lib/python3.8/site-packages/python_runner/python_action_wrapper.py\", line 395, in <module>
    obj.run()
  File \"/opt/stackstorm/st2/lib/python3.8/site-packages/python_runner/python_action_wrapper.py\", line 214, in run
    output = action.run(**self._parameters)
  File \"/opt/stackstorm/packs/st2/actions/actions_list.py\", line 23, in run
    result = self._run_client_method(method=self.client.actions.get_all,
  File \"/opt/stackstorm/packs/st2/actions/lib/action.py\", line 112, in _run_client_method
    result = method(**method_kwargs)
  File \"/opt/stackstorm/st2/lib/python3.8/site-packages/st2client/models/core.py\", line 45, in decorate
    return func(*args, **kwargs)
  File \"/opt/stackstorm/st2/lib/python3.8/site-packages/st2client/models/core.py\", line 247, in get_all
    response = self.client.get(url=url, params=params, **kwargs)
  File \"/opt/stackstorm/st2/lib/python3.8/site-packages/st2client/utils/httpclient.py\", line 33, in decorate
    return func(*args, **kwargs)
  File \"/opt/stackstorm/st2/lib/python3.8/site-packages/st2client/utils/httpclient.py\", line 80, in decorate
    return func(*args, **kwargs)
  File \"/opt/stackstorm/st2/lib/python3.8/site-packages/st2client/utils/httpclient.py\", line 61, in decorate
    return func(*args, **kwargs)
  File \"/opt/stackstorm/st2/lib/python3.8/site-packages/st2client/utils/httpclient.py\", line 120, in get
    response = requests.get(self.root + url, **kwargs)
  File \"/opt/stackstorm/st2/lib/python3.8/site-packages/requests/api.py\", line 76, in get
    return request('get', url, params=params, **kwargs)
  File \"/opt/stackstorm/st2/lib/python3.8/site-packages/requests/api.py\", line 61, in request
    return session.request(method=method, url=url, **kwargs)
  File \"/opt/stackstorm/st2/lib/python3.8/site-packages/requests/sessions.py\", line 542, in request
    resp = self.send(prep, **send_kwargs)
  File \"/opt/stackstorm/st2/lib/python3.8/site-packages/requests/sessions.py\", line 655, in send
    r = adapter.send(request, **kwargs)
  File \"/opt/stackstorm/st2/lib/python3.8/site-packages/requests/adapters.py\", line 516, in send
    raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPSConnectionPool(host='st2api.eog-prd-3-7.svc.cluster.local', port=9101): Max retries exceeded with url: /v1/actions?limit=10 (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7ffff63b1130>: Failed to establish a new connection: [Errno -2] Name or service not known'))
",
  "exit_code": 1,
  "result": "None"
}

But if I use postman to get the action list or st2actionrunner terminal I am not encounter this error. Why only I used this action I get the issue?

Same goes with registering the pack via packs.install
I will encountered below error

{
  "output": {
    "packs_list": [
      "st2"
    ],
    "message": "Successfully installed packs",
    "conflict_list": [],
    "warning_list": []
  },
  "errors": [
    {
      "type": "error",
      "message": "Execution failed. See result for details.",
      "task_id": "register_pack",
      "result": {
        "stdout": "",
        "stderr": "st2.actions.python.St2RegisterAction: DEBUG    Calling client method \"register\" with kwargs \"{'types': ['all'], 'packs': ['st2']}\"
Traceback (most recent call last):
  File \"/opt/stackstorm/st2/lib/python3.8/site-packages/urllib3/connection.py\", line 174, in _new_conn
    conn = connection.create_connection(
  File \"/opt/stackstorm/st2/lib/python3.8/site-packages/urllib3/util/connection.py\", line 72, in create_connection
    for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM):
  File \"/usr/lib/python3.8/socket.py\", line 918, in getaddrinfo
    for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
socket.gaierror: [Errno -2] Name or service not known

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File \"/opt/stackstorm/st2/lib/python3.8/site-packages/urllib3/connectionpool.py\", line 703, in urlopen
    httplib_response = self._make_request(
  File \"/opt/stackstorm/st2/lib/python3.8/site-packages/urllib3/connectionpool.py\", line 398, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File \"/opt/stackstorm/st2/lib/python3.8/site-packages/urllib3/connection.py\", line 239, in request
    super(HTTPConnection, self).request(method, url, body=body, headers=headers)
  File \"/usr/lib/python3.8/http/client.py\", line 1256, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File \"/usr/lib/python3.8/http/client.py\", line 1302, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File \"/usr/lib/python3.8/http/client.py\", line 1251, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File \"/usr/lib/python3.8/http/client.py\", line 1011, in _send_output
    self.send(msg)
  File \"/usr/lib/python3.8/http/client.py\", line 951, in send
    self.connect()
  File \"/opt/stackstorm/st2/lib/python3.8/site-packages/urllib3/connection.py\", line 205, in connect
    conn = self._new_conn()
  File \"/opt/stackstorm/st2/lib/python3.8/site-packages/urllib3/connection.py\", line 186, in _new_conn
    raise NewConnectionError(
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7ffff6335820>: Failed to establish a new connection: [Errno -2] Name or service not known

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File \"/opt/stackstorm/st2/lib/python3.8/site-packages/requests/adapters.py\", line 439, in send
    resp = conn.urlopen(
  File \"/opt/stackstorm/st2/lib/python3.8/site-packages/urllib3/connectionpool.py\", line 785, in urlopen
    retries = retries.increment(
  File \"/opt/stackstorm/st2/lib/python3.8/site-packages/urllib3/util/retry.py\", line 592, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='st2api.eog-prd-3-7.svc.cluster.local', port=9101): Max retries exceeded with url: /v1/packs/register (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ffff6335820>: Failed to establish a new connection: [Errno -2] Name or service not known'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File \"/opt/stackstorm/st2/lib/python3.8/site-packages/python_runner/python_action_wrapper.py\", line 395, in <module>
    obj.run()
  File \"/opt/stackstorm/st2/lib/python3.8/site-packages/python_runner/python_action_wrapper.py\", line 214, in run
    output = action.run(**self._parameters)
  File \"/opt/stackstorm/packs/packs/actions/pack_mgmt/register.py\", line 76, in run
    result = self._run_client_method(
  File \"/opt/stackstorm/packs/packs/actions/pack_mgmt/register.py\", line 155, in _run_client_method
    result = method(**method_kwargs)
  File \"/opt/stackstorm/st2/lib/python3.8/site-packages/st2client/models/core.py\", line 45, in decorate
    return func(*args, **kwargs)
  File \"/opt/stackstorm/st2/lib/python3.8/site-packages/st2client/models/core.py\", line 682, in register
    response = self.client.post(url, payload, **kwargs)
  File \"/opt/stackstorm/st2/lib/python3.8/site-packages/st2client/utils/httpclient.py\", line 33, in decorate
    return func(*args, **kwargs)
  File \"/opt/stackstorm/st2/lib/python3.8/site-packages/st2client/utils/httpclient.py\", line 80, in decorate
    return func(*args, **kwargs)
  File \"/opt/stackstorm/st2/lib/python3.8/site-packages/st2client/utils/httpclient.py\", line 91, in decorate
    return func(*args, **kwargs)
  File \"/opt/stackstorm/st2/lib/python3.8/site-packages/st2client/utils/httpclient.py\", line 61, in decorate
    return func(*args, **kwargs)
  File \"/opt/stackstorm/st2/lib/python3.8/site-packages/st2client/utils/httpclient.py\", line 129, in post
    response = requests.post(self.root + url, json.dumps(data), **kwargs)
  File \"/opt/stackstorm/st2/lib/python3.8/site-packages/requests/api.py\", line 119, in post
    return request('post', url, data=data, json=json, **kwargs)
  File \"/opt/stackstorm/st2/lib/python3.8/site-packages/requests/api.py\", line 61, in request
    return session.request(method=method, url=url, **kwargs)
  File \"/opt/stackstorm/st2/lib/python3.8/site-packages/requests/sessions.py\", line 542, in request
    resp = self.send(prep, **send_kwargs)
  File \"/opt/stackstorm/st2/lib/python3.8/site-packages/requests/sessions.py\", line 655, in send
    r = adapter.send(request, **kwargs)
  File \"/opt/stackstorm/st2/lib/python3.8/site-packages/requests/adapters.py\", line 516, in send
    raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPConnectionPool(host='st2api.eog-prd-3-7.svc.cluster.local', port=9101): Max retries exceeded with url: /v1/packs/register (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ffff6335820>: Failed to establish a new connection: [Errno -2] Name or service not known'))
",
        "exit_code": 1,
        "result": "None"
      }
    }
  ]
}

[arm4b]

The relevant error is this:

socket.gaierror: [Errno -2] Name or service not known

The issue here is DNS lookup that's failing in your K8s cluster and so stackstorm can't connect to st2api.
Try to debug the DNS in your K8s cluster and make sure it works.