Problem installing or searching pack, temporary failure in name resolution #216

JustDevZero · 2021-02-26T00:10:13Z

Describe the problem
A clear and concise description of what the problem is.

Versions

Host OS: Fedora 33 And Ubuntu 18.04
docker: Docker version 20.10.3, build 48d30b5
docker-compose: docker-compose version 1.28.2, build 67630359
stackstorm version: v3.3.0

To Reproduce
If necessary, please specify steps to reproduce the behavior.

Login to st2client with:

docker-compose exec st2client bash

And then try to search or install.

It will fail with:


st2 pack search datadog

root@554d99419ce3:/opt/stackstorm# st2 pack search docker
ERROR: 400 Client Error: Bad Request
MESSAGE: No results from the index: tried https://index.stackstorm.org/v1/index.json.
Status: [
    {
        "url": "https://index.stackstorm.org/v1/index.json",
        "packs": 0,
        "message": "ConnectionError(MaxRetryError(\"HTTPSConnectionPool(host='index.stackstorm.org', port=443): Max retries exceeded with url: /v1/index.json (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f448e565710>: Failed to establish a new connection: [Errno -3] Lookup timed out',))\",),)",
        "error": "unresponsive"
    }
] for url: http://st2api:9101/packs/index/search



root@554d99419ce3:/opt/stackstorm# st2 pack install datadog

	[ succeeded ] init_task
	[  failed   ] download_pack

id: 6038397af6cde4fdb05d5c08
action.ref: packs.install
parameters: 
  packs:
  - datadog
  python3: false
status: failed
start_timestamp: Thu, 25 Feb 2021 23:57:46 UTC
end_timestamp: Thu, 25 Feb 2021 23:57:59 UTC
result: 
  errors:
  - message: Execution failed. See result for details.
    result:
      exit_code: 1
      result: None
      stderr: "Index parsing error: {
    "url": "https://index.stackstorm.org/v1/index.json",
    "packs": 0,
    "message": "ConnectionError(MaxRetryError(\\"HTTPSConnectionPool(host='index.stackstorm.org', port=443): Max retries exceeded with url: /v1/index.json (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7ff8ff5cdc18>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution',))\\",),)",
    "error": "unresponsive"
}
Traceback (most recent call last):
  File "/opt/stackstorm/st2/lib/python3.6/site-packages/python_runner/python_action_wrapper.py", line 334, in <module>
    obj.run()
  File "/opt/stackstorm/st2/lib/python3.6/site-packages/python_runner/python_action_wrapper.py", line 193, in run
    output = action.run(**self._parameters)
  File "/opt/stackstorm/packs/packs/actions/pack_mgmt/download.py", line 90, in run
    return self._validate_result(result=result, repo_url=pack_url)
  File "/opt/stackstorm/packs/packs/actions/pack_mgmt/download.py", line 111, in _validate_result
    raise Exception(message)
Exception: The pack has not been downloaded from "None".

Errors:
No results from the index: tried https://index.stackstorm.org/v1/index.json.
Status: [
    {
        "url": "https://index.stackstorm.org/v1/index.json",
        "packs": 0,
        "message": "ConnectionError(MaxRetryError(\\"HTTPSConnectionPool(host='index.stackstorm.org', port=443): Max retries exceeded with url: /v1/index.json (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7ff8ff5cdc18>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution',))\\",),)",
        "error": "unresponsive"
    }
]
"
      stdout: ''
    task_id: download_pack
    type: error
  output:
    conflict_list: null
    message: ''
    packs_list: null
    warning_list: null
+--------------------------+------------------------+---------------+----------------+-------------------------------+
| id                       | status                 | task          | action         | start_timestamp               |
+--------------------------+------------------------+---------------+----------------+-------------------------------+
| 6038397b1ac1577e7e12dae3 | succeeded (0s elapsed) | init_task     | core.noop      | Thu, 25 Feb 2021 23:57:47 UTC |
| 6038397c1ac1577e7e12daf3 | failed (11s elapsed)   | download_pack | packs.download | Thu, 25 Feb 2021 23:57:48 UTC |

If try to do some curl... from st2api container:


root@2bc662f52cab:/opt/stackstorm# curl https://index.stackstorm.org/v1/index.json|wc -l
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:--  0:00:09 --:--:--     0curl: (6) Could not resolve host: index.stackstorm.org
0

Modifying resolve.conf for adding 1.1.1.1 after docker resolution improves the thing but still.


curl https://index.stackstorm.org/v1/index.json|wc -l
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  678k    0  678k    0     0   406k      0 --:--:--  0:00:02 --:--:--  406k
16298

From s2client container on the other hand it's inmediate.

curl https://index.stackstorm.org/v1/index.json|wc -l
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  678k    0  678k    0     0  3879k      0 --:--:-- --:--:-- --:--:-- 3857k
16298

For some reason, s2api container it's taking more time to resolv... But even with that it's no reason for getting a Temporary failure in name resolution.

Am i installing them wrong?

The text was updated successfully, but these errors were encountered:

minsis · 2021-03-04T21:19:51Z

By default bridge networks dont forward to the outside world and need to have additional parameters set on the host for this to work correctly.

https://docs.docker.com/network/bridge/#enable-forwarding-from-docker-containers-to-the-outside-world

JustDevZero · 2021-03-04T21:31:36Z

@minsis I tried it, and indeed also tried changing the type of network, but the problem is still the same.

minsis · 2021-03-04T21:34:19Z

@JustDevZero We also faced some of these issues (not with StackStorm) when ipv6 was enabled and ultimately disabled it. Do you also have ipv6 enabled?

JustDevZero · 2021-03-04T21:37:28Z

I'll check it as soon as I'm at my office hours, but I think you're maybe having a point on the ipv6, AFAIK we have ipv6 disabled on whole network. El jue., 4 mar. 2021 22:34, Dennis Whitney <[email protected]> escribió:

…

@JustDevZero <https://github.com/JustDevZero> We also faced some of these issues (not with StackStorm) when ipv6 was enabled and ultimately disabled it. Do you also have ipv6 enabled? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#216 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AABKFV33A6MSV3UXJOM4ZJTTB74GVANCNFSM4YHOXTIQ> .

minsis · 2021-03-04T21:43:41Z

@JustDevZero The network itself can have ipv6 disabled as a whole but that doesn't prevent localhost from having it enabled. Every device has its own link local ipv6 address when it comes up.

JustDevZero · 2021-03-08T11:57:17Z

It looks that it's ipv6 fault then, host have ipv6 disabled but somehow docker it's still trying to get ipv6 not 4. So... I guess it's matter of dealing with docker config, maybe?

daniel-rocha-ciandt · 2021-09-11T21:25:05Z

Hi,
I'm also facing the same issue, from curl it works perfectly.

Every action in python 3 that need to resolve a DNS happens the same problem:

@JustDevZero have you manager to find out what was the root cause of this issue?

Versions

Host OS: Ubuntu 20.04
docker: Docker version 20.10.7, build 20.10.7-0ubuntu1~20.04.1
docker-compose version 1.29.1, build c34c88b2
stackstorm version: v3.5.0
Python: v3.6.9

cydergoth · 2021-10-12T18:32:35Z

This fixed the issue for me:

diff --git a/docker-compose.yml b/docker-compose.yml
index 3a05ac0..b1c47fd 100644
--- a/docker-compose.yml
+++ b/docker-compose.yml
@@ -2,6 +2,9 @@ version: '3'

 services:
   st2web:
+    dns:
+      - 8.8.8.8
+      - 4.4.4.4
     image: ${ST2_IMAGE_REPO:-stackstorm/}st2web:${ST2_VERSION:-latest}
     restart: on-failure
     environment:
@@ -42,6 +45,9 @@ services:
     dns_search: .
     command: /makesecrets.sh
   st2api:
+    dns:
+      - 8.8.8.8
+      - 4.4.4.4
     image: ${ST2_IMAGE_REPO:-stackstorm/}st2api:${ST2_VERSION:-latest}
     restart: on-failure
     depends_on:
@@ -65,6 +71,9 @@ services:
       - ${ST2_PACKS_DEV:-./packs.dev}:/opt/stackstorm/packs.dev:rw
     dns_search: .
   st2stream:
+    dns:
+      - 8.8.8.8
+      - 4.4.4.4
     image: ${ST2_IMAGE_REPO:-stackstorm/}st2stream:${ST2_VERSION:-latest}
     restart: on-failure
     depends_on:
@@ -76,6 +85,9 @@ services:
       - ./files/st2.user.conf:/etc/st2/st2.user.conf:ro
     dns_search: .
   st2scheduler:
+    dns:
+      - 8.8.8.8
+      - 4.4.4.4
     image: ${ST2_IMAGE_REPO:-stackstorm/}st2scheduler:${ST2_VERSION:-latest}
     restart: on-failure
     depends_on:
@@ -88,6 +100,9 @@ services:
       - ./files/st2.user.conf:/etc/st2/st2.user.conf:ro
     dns_search: .
   st2workflowengine:
+    dns:
+      - 8.8.8.8
+      - 4.4.4.4
     image: ${ST2_IMAGE_REPO:-stackstorm/}st2workflowengine:${ST2_VERSION:-latest}
     restart: on-failure
     depends_on:
@@ -101,6 +116,9 @@ services:
       - stackstorm-keys:/etc/st2/keys:ro
     dns_search: .
   st2auth:
+    dns:
+      - 8.8.8.8
+      - 4.4.4.4
     image: ${ST2_IMAGE_REPO:-stackstorm/}st2auth:${ST2_VERSION:-latest}
     restart: on-failure
     depends_on:
@@ -113,6 +131,9 @@ services:
       - ./files/htpasswd:/etc/st2/htpasswd:ro
     dns_search: .
   st2actionrunner:
+    dns:
+      - 8.8.8.8
+      - 4.4.4.4
     image: ${ST2_IMAGE_REPO:-stackstorm/}st2actionrunner:${ST2_VERSION:-latest}
     restart: on-failure
     depends_on:
@@ -133,6 +154,9 @@ services:
       - stackstorm-keys:/etc/st2/keys:ro
     dns_search: .
   st2garbagecollector:
+    dns:
+      - 8.8.8.8
+      - 4.4.4.4
     image: ${ST2_IMAGE_REPO:-stackstorm/}st2garbagecollector:${ST2_VERSION:-latest}
     restart: on-failure
     depends_on:
@@ -144,6 +168,9 @@ services:
       - ./files/st2.user.conf:/etc/st2/st2.user.conf:ro
     dns_search: .
   st2notifier:
+    dns:
+      - 8.8.8.8
+      - 4.4.4.4
     image: ${ST2_IMAGE_REPO:-stackstorm/}st2notifier:${ST2_VERSION:-latest}
     restart: on-failure
     depends_on:
@@ -156,6 +183,9 @@ services:
       - ./files/st2.user.conf:/etc/st2/st2.user.conf:ro
     dns_search: .
   st2rulesengine:
+    dns:
+      - 8.8.8.8
+      - 4.4.4.4
     image: ${ST2_IMAGE_REPO:-stackstorm/}st2rulesengine:${ST2_VERSION:-latest}
     restart: on-failure
     depends_on:
@@ -167,6 +197,9 @@ services:
       - ./files/st2.user.conf:/etc/st2/st2.user.conf:ro
     dns_search: .
   st2sensorcontainer:
+    dns:
+      - 8.8.8.8
+      - 4.4.4.4
     image: ${ST2_IMAGE_REPO:-stackstorm/}st2sensorcontainer:${ST2_VERSION:-latest}
     restart: on-failure
     depends_on:
@@ -182,6 +215,9 @@ services:
       - stackstorm-packs-configs:/opt/stackstorm/configs:ro
       - ${ST2_PACKS_DEV:-./packs.dev}:/opt/stackstorm/packs.dev:ro
   st2timersengine:
+    dns:
+      - 8.8.8.8
+      - 4.4.4.4
     image: ${ST2_IMAGE_REPO:-stackstorm/}st2timersengine:${ST2_VERSION:-latest}
     restart: on-failure
     depends_on:
@@ -192,6 +228,9 @@ services:
     volumes:
       - ./files/st2.docker.conf:/etc/st2/st2.docker.conf:ro
   st2client:
+    dns:
+      - 8.8.8.8
+      - 4.4.4.4
     image: ${ST2_IMAGE_REPO:-stackstorm/}st2actionrunner:${ST2_VERSION:-latest}
     restart: on-failure
     depends_on:
@@ -222,6 +261,9 @@ services:
       - ./files/st2-cli.conf:/root/.st2/config
       - ./scripts/st2client-startup.sh:/st2client-startup.sh
   st2chatops:
+    dns:
+      - 8.8.8.8
+      - 4.4.4.4
     image: ${ST2_IMAGE_REPO:-stackstorm/}st2chatops:${ST2_VERSION:-latest}
     restart: on-failure:5
     depends_on:

minsis · 2021-10-12T18:38:18Z

By default ipv4 forwarding is not enabled. This is mostly where the issue is with getting traffic to leave the container.

Add net.ipv4.ip_forward=1 to /etc/sysctl.conf to fix it.

arm4b added the question label Mar 5, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Problem installing or searching pack, temporary failure in name resolution #216

Problem installing or searching pack, temporary failure in name resolution #216

JustDevZero commented Feb 26, 2021 •

edited

Loading

minsis commented Mar 4, 2021

JustDevZero commented Mar 4, 2021

minsis commented Mar 4, 2021

JustDevZero commented Mar 4, 2021 via email

minsis commented Mar 4, 2021

JustDevZero commented Mar 8, 2021

daniel-rocha-ciandt commented Sep 11, 2021

cydergoth commented Oct 12, 2021 •

edited

Loading

minsis commented Oct 12, 2021

Problem installing or searching pack, temporary failure in name resolution #216

Problem installing or searching pack, temporary failure in name resolution #216

Comments

JustDevZero commented Feb 26, 2021 • edited Loading

minsis commented Mar 4, 2021

JustDevZero commented Mar 4, 2021

minsis commented Mar 4, 2021

JustDevZero commented Mar 4, 2021 via email

minsis commented Mar 4, 2021

JustDevZero commented Mar 8, 2021

daniel-rocha-ciandt commented Sep 11, 2021

cydergoth commented Oct 12, 2021 • edited Loading

minsis commented Oct 12, 2021

JustDevZero commented Feb 26, 2021 •

edited

Loading

cydergoth commented Oct 12, 2021 •

edited

Loading