-
Notifications
You must be signed in to change notification settings - Fork 1
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
- Loading branch information
Showing
13 changed files
with
858 additions
and
1 deletion.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,101 @@ | ||
| <!DOCTYPE html> | ||
| <html lang="en"> | ||
| <head> | ||
| <meta charset="UTF-8"> | ||
| <meta name="description" content="This post records how I debug a docker issue"> | ||
| <link rel="alternate" | ||
| type="application/rss+xml" | ||
| href="https://chenyo-17.github.io/org-static-blog/rss.xml" | ||
| title="RSS feed for https://chenyo-17.github.io/org-static-blog"> | ||
| <title>A stupid debugging experience</title> | ||
| </head> | ||
| <body> | ||
| <div id="preamble" class="status"></div> | ||
| <div id="content"> | ||
| <div class="post-date">24 Jun 2024</div><h1 class="post-title"><a href="https://chenyo-17.github.io/org-static-blog/2024-06-24-a-stupid-debugging-experience.html">A stupid debugging experience</a></h1> | ||
| <nav id="table-of-contents" role="doc-toc"> | ||
| <h2>Table of Contents</h2> | ||
| <div id="text-table-of-contents" role="doc-toc"> | ||
| <ul> | ||
| <li><a href="#org9cc078c">1. What happended</a></li> | ||
| <li><a href="#orgfd435b4">2. What did I do</a></li> | ||
| <li><a href="#org436bc46">3. Another issue of running RPC in docker</a></li> | ||
| </ul> | ||
| </div> | ||
| </nav> | ||
| <div id="outline-container-org9cc078c" class="outline-2"> | ||
| <h2 id="org9cc078c"><span class="section-number-2">1.</span> What happended</h2> | ||
| <div class="outline-text-2" id="text-1"> | ||
| <ul class="org-ul"> | ||
| <li>Servers SA and SB have the same docker installation, and the same running container CA and CB.</li> | ||
| <li><p> | ||
| A Go file G can be built on CA, but on CB it reports this error: | ||
| </p> | ||
| <blockquote> | ||
| <p> | ||
| runtime: failed to create new OS thread (have 2 already; errno=11) | ||
| runtime: may need to increase max user processes (ulimit -u) | ||
| fatal error: newosproc" | ||
| </p> | ||
| </blockquote></li> | ||
| </ul> | ||
| </div> | ||
| </div> | ||
| <div id="outline-container-orgfd435b4" class="outline-2"> | ||
| <h2 id="orgfd435b4"><span class="section-number-2">2.</span> What did I do</h2> | ||
| <div class="outline-text-2" id="text-2"> | ||
| <ol class="org-ol"> | ||
| <li>I compared any related configurations between SA and SB. and between CA and CB, e.g., <code class="src src-bash"><span style="color: #c678dd;">ulimit</span> -a</code>, <code class="src src-bash">/etc/security/limits.conf</code>. They all look the same.</li> | ||
| <li>I created a new container CN on SA with the same docker image, CN can compile G.</li> | ||
| <li>I looked into the (complex) `docker run` script for CA/CB and figured out it was due to a resource constraint `–pids-limit 100`. | ||
| <ul class="org-ul"> | ||
| <li>Increasing this limit to 200 seems resolve the issue, but I had no idea why the Go compiler needed so many resources (perhaps due to package I imported).</li> | ||
| </ul></li> | ||
| <li><b><b>Until now</b></b>, I realized, since the container did not support the compilation, why not just only transfer the compiled binary! | ||
| <ul class="org-ul"> | ||
| <li>How silly that I didn’t even try this in the beginning!</li> | ||
| </ul></li> | ||
| <li>Since the program imports the `net` package, and there is a <a href="https://www.reddit.com/r/golang/comments/pi97sp/what_is_the_consequence_of_using_cgo_enabled0/">known issue</a> of Alpine image running a Go binary file, I followed the post and disabled `CGO`on SA, then `docker cp` the binary to CA, and it worked.</li> | ||
| </ol> | ||
| </div> | ||
| </div> | ||
| <div id="outline-container-org436bc46" class="outline-2"> | ||
| <h2 id="org436bc46"><span class="section-number-2">3.</span> Another issue of running RPC in docker</h2> | ||
| <div class="outline-text-2" id="text-3"> | ||
| <ul class="org-ul"> | ||
| <li>The other day, I also spent hours debugging a `route unreachable` error when I want to send a request from CA to SA.</li> | ||
| <li>The CA is using the `bridge` network, so it should talk to SA via SA’s interface `docker0` within the subnet `172.17.0.0/16`.</li> | ||
| <li><p> | ||
| However, in my case, the docker by default rejects packages from any container as shown in SA’s `tcpdump` result: | ||
| </p> | ||
| <blockquote> | ||
| <p> | ||
| 172.17.0.1->172.17.0.3 ICMP host unreachable- admin prohibited, length 68 | ||
| </p> | ||
| </blockquote></li> | ||
|
|
||
| <li><p> | ||
| By checking SA’s iptables, I found this rule: | ||
| </p> | ||
| <div class="org-src-container"> | ||
| <pre class="src src-bash"> -A INPUT -j REJECT --reject-with icmp-host-prohibited | ||
| </pre> | ||
| </div> | ||
| <ul class="org-ul"> | ||
| <li>Strangely, the `ping` still works with this rule.</li> | ||
| </ul></li> | ||
|
|
||
| <li><p> | ||
| In the end, I need to append a new rule to make the RPC work. | ||
| </p> | ||
| <div class="org-src-container"> | ||
| <pre class="src src-bash"> iptables -I INPUT <span style="color: #da8548; font-weight: bold;">1</span> -i docker0 -p tcp --dport <port> -s 172.17.0.0/16 -j ACCEPT | ||
| </pre> | ||
| </div></li> | ||
| </ul> | ||
| </div> | ||
| </div> | ||
| <div class="taglist"><a href="https://chenyo-17.github.io/org-static-blog/tags.html">Tags</a>: <a href="https://chenyo-17.github.io/org-static-blog/tag-tools:docker.html">tools:docker</a> <a href="https://chenyo-17.github.io/org-static-blog/tag-lang:go.html">lang:go</a> <a href="https://chenyo-17.github.io/org-static-blog/tag-tools:linux:apline.html">tools:linux:apline</a> </div></div> | ||
| <div id="postamble" class="status"></div> | ||
| </body> | ||
| </html> |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,45 @@ | ||
| <!DOCTYPE html> | ||
| <html lang="en"> | ||
| <head> | ||
| <meta charset="UTF-8"> | ||
| <meta name="description" content="This post records some linux use tips"> | ||
| <link rel="alternate" | ||
| type="application/rss+xml" | ||
| href="https://chenyo-17.github.io/org-static-blog/rss.xml" | ||
| title="RSS feed for https://chenyo-17.github.io/org-static-blog"> | ||
| <title>Linux use tips</title> | ||
| </head> | ||
| <body> | ||
| <div id="preamble" class="status"></div> | ||
| <div id="content"> | ||
| <div class="post-date">24 Jun 2024</div><h1 class="post-title"><a href="https://chenyo-17.github.io/org-static-blog/2024-06-24-linux-use-tips.html">Linux use tips</a></h1> | ||
| <nav id="table-of-contents" role="doc-toc"> | ||
| <h2>Table of Contents</h2> | ||
| <div id="text-table-of-contents" role="doc-toc"> | ||
| <ul> | ||
| <li><a href="#orgc8c691f">1. i3</a> | ||
| <ul> | ||
| <li><a href="#orgba542b4">1.1. move specific workspaces between different monitors (ref)</a></li> | ||
| </ul> | ||
| </li> | ||
| </ul> | ||
| </div> | ||
| </nav> | ||
| <div id="outline-container-orgc8c691f" class="outline-2"> | ||
| <h2 id="orgc8c691f"><span class="section-number-2">1.</span> i3</h2> | ||
| <div class="outline-text-2" id="text-1"> | ||
| </div> | ||
| <div id="outline-container-orgba542b4" class="outline-3"> | ||
| <h3 id="orgba542b4"><span class="section-number-3">1.1.</span> move specific workspaces between different monitors <a href="https://i3wm.org/docs/user-contributed/swapping-workspaces.html">(ref)</a></h3> | ||
| <div class="outline-text-3" id="text-1-1"> | ||
| <ol class="org-ol"> | ||
| <li>adjust the monitor relative positions</li> | ||
| <li>use <code class="src src-bash">i3-msg -- move workspace to output right</code> to move the <b><b>current</b></b> workspace to the monitor on the right</li> | ||
| </ol> | ||
| </div> | ||
| </div> | ||
| </div> | ||
| <div class="taglist"><a href="https://chenyo-17.github.io/org-static-blog/tags.html">Tags</a>: <a href="https://chenyo-17.github.io/org-static-blog/tag-tools:linux:arch.html">tools:linux:arch</a> <a href="https://chenyo-17.github.io/org-static-blog/tag-tools:linux:i3.html">tools:linux:i3</a> </div></div> | ||
| <div id="postamble" class="status"></div> | ||
| </body> | ||
| </html> |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,41 @@ | ||
| #+title: A stupid debugging experience | ||
| #+date: <2024-06-24 15:06> | ||
| #+description: This post records how I debug a docker issue | ||
| #+filetags: tools:docker lang:go tools:linux:apline | ||
|
|
||
| * What happended | ||
| - Servers SA and SB have the same docker installation, and the same running container CA and CB. | ||
| - A Go file G can be built on CA, but on CB it reports this error: | ||
| #+begin_quote | ||
| runtime: failed to create new OS thread (have 2 already; errno=11) | ||
| runtime: may need to increase max user processes (ulimit -u) | ||
| fatal error: newosproc" | ||
| #+end_quote | ||
|
|
||
| * What did I do | ||
| 1. I compared any related configurations between SA and SB. and between CA and CB, e.g., src_bash[:exports code]{ulimit -a}, src_bash[:exports code]{/etc/security/limits.conf}. They all look the same. | ||
| 2. I created a new container CN on SA with the same docker image, CN can compile G. | ||
| 3. I looked into the (complex) ~docker run~ script for CA/CB and figured out it was due to a resource constraint ~--pids-limit 100~. | ||
| - Increasing this limit to 200 seems resolve the issue, but I had no idea why the Go compiler needed so many resources (perhaps due to package I imported). | ||
| 4. **Until this point**, I realized, since the container did not support the compilation, why not just only transfer the compiled binary! | ||
| - How silly that I didn't even try this in the beginning! | ||
| 5. Since the program imports the ~net~ package, and there is a [[https://www.reddit.com/r/golang/comments/pi97sp/what_is_the_consequence_of_using_cgo_enabled0/][known issue]] of Alpine image running a Go binary file, I followed the post and disabled ~CGO~ on SA, then ~docker cp~ the binary to CA, and it worked. | ||
|
|
||
| * Another issue of running RPC in docker | ||
| - The other day, I also spent hours debugging a `route unreachable` error when I want to send a request from CA to SA. | ||
| - The CA is using the ~bridge~ network, so it should talk to SA via SA's interface ~docker0~ within the subnet ~172.17.0.0/16~. | ||
| - However, in my case, the docker by default rejects packages from any container as shown in SA's ~tcpdump~ result: | ||
| #+begin_quote | ||
| 172.17.0.1->172.17.0.3 ICMP host unreachable- admin prohibited, length 68 | ||
| #+end_quote | ||
|
|
||
| - By checking SA's iptables, I found this rule: | ||
| #+begin_src bash | ||
| -A INPUT -j REJECT --reject-with icmp-host-prohibited | ||
| #+end_src | ||
| - Strangely, the ~ping~ still works with this rule. | ||
|
|
||
| - In the end, I need to append a new rule to make the RPC work. | ||
| #+begin_src bash | ||
| iptables -I INPUT 1 -i docker0 -p tcp --dport <port> -s 172.17.0.0/16 -j ACCEPT | ||
| #+end_src |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,9 @@ | ||
| #+title: Linux use tips | ||
| #+date: <2024-06-24 09:17> | ||
| #+description: This post records some linux use tips | ||
| #+filetags: tools:linux:arch tools:linux:i3 | ||
|
|
||
| * i3 | ||
| ** move specific workspaces between different monitors [[https://i3wm.org/docs/user-contributed/swapping-workspaces.html][(ref)]] | ||
| 1. adjust the monitor relative positions | ||
| 2. use src_bash[:exports code]{i3-msg -- move workspace to output right} to move the **current** workspace to the monitor on the right |
Oops, something went wrong.