-
Notifications
You must be signed in to change notification settings - Fork 1
/
2024-06-24-a-stupid-debugging-experience.html
139 lines (137 loc) · 6.59 KB
/
2024-06-24-a-stupid-debugging-experience.html
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<meta name="description" content="This post records how I debug a docker issue">
<link rel="alternate"
type="application/rss+xml"
href="https://chenyo.me/rss.xml"
title="RSS feed for https://chenyo.me">
<title>A stupid debugging experience</title>
<script type="text/javascript"
src="https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.1/MathJax.js?config=TeX-AMS-MML_HTMLorMML">
</script>
<script type="text/x-mathjax-config">
MathJax.Hub.Config({tex2jax: {inlineMath: [['$','$'],['\\(','\\)']]}});
</script>
<meta name="author" content="chenyo">
<meta name="referrer" content="no-referrer">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<link rel="stylesheet" href="assets/style.css" type="text/css"/>
<link rel="stylesheet" href="https://cdnjs.cloudflare.com/ajax/libs/font-awesome/6.4.0/css/all.min.css"/>
<link rel="favicon" type="image/x-icon" href="favicon.ico">
<script src="assets/search.js"></script></head>
<body>
<div id="preamble" class="status">
<header>
<h1><a href="https://chenyo.me">Chenyo's org-static-blog</a></h1>
<nav>
<a href="https://chenyo.me">Home</a>
<a href="archive.html">Archive</a>
<a href="tags.html">Tags</a>
<div id="search-container">
<input type="text" id="search-input" placeholder="Search anywhere...">
<i class="fas fa-search search-icon"></i>
</div>
</nav>
</header></div>
<div id="content">
<div class="post-date">24 Jun 2024</div><h1 class="post-title"><a href="https://chenyo.me/2024-06-24-a-stupid-debugging-experience.html">A stupid debugging experience</a></h1>
<nav id="table-of-contents" role="doc-toc">
<h2>Table of Contents</h2>
<div id="text-table-of-contents" role="doc-toc">
<ul>
<li><a href="#orgb8fb5fc">1. What happened</a></li>
<li><a href="#orgebf733a">2. What did I do</a></li>
<li><a href="#org98115f4">3. Another issue of running RPC in docker</a></li>
</ul>
</div>
</nav>
<div id="outline-container-orgb8fb5fc" class="outline-2">
<h2 id="orgb8fb5fc"><span class="section-number-2">1.</span> What happened</h2>
<div class="outline-text-2" id="text-1">
<ul class="org-ul">
<li>Servers SA and SB have the same docker installation, and the same running container CA and CB.</li>
<li><p>
A Go file G can be built on CA, but on CB it reports this error:
</p>
<p class="verse">
runtime: failed to create new OS thread (have 2 already; errno=11)<br>
runtime: may need to increase max user processes (ulimit -u)<br>
fatal error: newosproc<br>
</p></li>
</ul>
</div>
</div>
<div id="outline-container-orgebf733a" class="outline-2">
<h2 id="orgebf733a"><span class="section-number-2">2.</span> What did I do</h2>
<div class="outline-text-2" id="text-2">
<ol class="org-ol">
<li>I compared any related configurations between SA and SB. and between CA and CB, e.g., <code class="src src-bash"><span style="color: #c678dd;">ulimit</span> -a</code>, <code class="src src-bash">/etc/security/limits.conf</code>. They all look the same.</li>
<li>I created a new container CN on SA with the same docker image, CN can compile G.</li>
<li>I looked into the (complex) <code>docker run</code> script for CA/CB and figured out it was due to a resource constraint <code>--pids-limit 100</code>.
<ul class="org-ul">
<li>Increasing this limit to 200 seems resolve the issue, but I had no idea why the Go compiler needed so many resources (perhaps due to package I imported).</li>
</ul></li>
<li><b><b>Until this point</b></b>, I realized, since the container did not support the compilation, why not just only transfer the compiled binary!
<ul class="org-ul">
<li>How silly that I didn’t even try this in the beginning!</li>
</ul></li>
<li>Since the program imports the <code>net</code> package, and there is a <a href="https://www.reddit.com/r/golang/comments/pi97sp/what_is_the_consequence_of_using_cgo_enabled0/">known issue</a> of Alpine image running a Go binary file, I followed the post and disabled <code>CGO</code> on SA, then <code>docker cp</code> the binary to CA, and it worked.</li>
</ol>
</div>
</div>
<div id="outline-container-org98115f4" class="outline-2">
<h2 id="org98115f4"><span class="section-number-2">3.</span> Another issue of running RPC in docker</h2>
<div class="outline-text-2" id="text-3">
<ul class="org-ul">
<li>The other day, I also spent hours debugging a <code>route unreachable</code> error when I want to send a request from CA to SA.</li>
<li>The CA is using the <code>bridge</code> network, so it should talk to SA via SA’s interface <code>docker0</code> within the subnet <code>172.17.0.0/16</code>.</li>
<li><p>
However, in my case, the docker by default rejects packages from any container as shown in SA’s <code>tcpdump</code> result:
</p>
<p class="verse">
172.17.0.1->172.17.0.3 ICMP host unreachable- admin prohibited, length 68<br>
</p></li>
<li><p>
By checking SA’s iptables, I found this rule:
</p>
<div class="org-src-container">
<pre class="src src-bash"> -A INPUT -j REJECT --reject-with icmp-host-prohibited
</pre>
</div>
<ul class="org-ul">
<li>Strangely, the <code>ping</code> still works with this rule.</li>
</ul></li>
<li><p>
In the end, I need to append a new rule to make the RPC work.
</p>
<div class="org-src-container">
<pre class="src src-bash"> iptables -I INPUT <span style="color: #da8548; font-weight: bold;">1</span> -i docker0 -p tcp --dport <port> -s 172.17.0.0/16 -j ACCEPT
</pre>
</div></li>
</ul>
</div>
</div>
<div class="taglist"><a href="https://chenyo.me/tags.html">Tags</a>: <a href="https://chenyo.me/tag-docker.html">docker</a> <a href="https://chenyo.me/tag-go.html">go</a> <a href="https://chenyo.me/tag-linux.html">linux</a> <a href="https://chenyo.me/tag-alpine.html">alpine</a> </div></div>
<div id="postamble" class="status"><div id="search-results"></div>
<footer>
<div class="footer-content">
<div class="footer-left">
<p>© 2024 chenyo. Some rights reserved.</p>
<a rel="license" href="http://creativecommons.org/licenses/by-nc/4.0/">
<img alt="Creative Commons License" style="border-width: 0"
src="https://i.creativecommons.org/l/by-nc/4.0/88x31.png"/>
</a>
</div>
<div class="social-links">
<a href="https://t.me/chenyo17" target="_blank" rel="noopener noreferrer">
<i class="fab fa-telegram"></i>
</a>
<a href="https://github.com/chenyo-17" target="_blank" rel="noopener noreferrer">
<i class="fab fa-github"></i>
</a>
</div>
</footer></div>
</body>
</html>