-
Notifications
You must be signed in to change notification settings - Fork 0
/
Copy pathlinux_application_tracing.txt
317 lines (264 loc) · 12.4 KB
/
linux_application_tracing.txt
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
[[{monitoring.jobs]]
# Tracing (Application&Kernel fine grined monitoring)
[[{profiling,security.encryption]]
[[dev_stack.mysql,dev_stack.postgresql,dev_stack.java]]
## Dynamic Tracing Tools
* Extracted from "Best linux monitoring tools" (<https://www.tecmint.com/bcc-best-linux-performance-monitoring-tools/>)
[[{monitoring.network.TLS]]
### sslsniff
<https://github.com/iovisor/bcc/blob/master/tools/sslsniff_example.txt>.
Trace write/send and read/recv functions of OpenSSL, GnuTLS and NSS.<br/>
Data passed to this functions is printed as plain text.
Useful, for example, to sniff HTTP before encrypted with SSL.
See also:
<https://github.com/tldr-pages/tldr/blob/master/pages/common/httpflow.md>
[[monitoring.network.TLS}]]
[[{monitoring.network.tcpconnect]]
### tcpconnect
* <https://github.com/iovisor/bcc/blob/master/tools/tcpconnect_example.txt>
* trace kernel function performing active TCP connections
(eg, via a connect() syscall; accept() are passive connections).
Ex:
```
| $ sudo tcpconnect
| PID COMM IP SADDR DADDR DPORT
| 1479 telnet 4 127.0.0.1 127.0.0.1 23
| 1469 curl 4 10.201.219.236 54.245.105.25 80
| 1469 curl 4 10.201.219.236 54.67.101.145 80
| 1991 telnet 6 ::1 ::1 23
| 2015 ssh 6 fe80::2000:bff:fe82:3ac fe80::2000:bff:fe82:3ac 22
```
[[monitoring.network.tcpconnect}]]
[[{monitoring.network.tcplife]]
### tcplife
* <https://github.com/iovisor/bcc/blob/master/tools/tcplife_example.txt>
summarizes TCP sessions that open and close while tracing. Ex:
```
| # ./tcplife
| PID COMM LADDR LPORT RADDR RPORT TX_KB RX_KB MS
| 22597 recordProg 127.0.0.1 46644 127.0.0.1 28527 0 0 0.23
| 3277 redis-serv 127.0.0.1 28527 127.0.0.1 46644 0 0 0.28
| 22598 curl 100.66.3.172 61620 52.205.89.26 80 0 1 91.79
| 22604 curl 100.66.3.172 44400 52.204.43.121 80 0 1 121.38
| 22624 recordProg 127.0.0.1 46648 127.0.0.1 28527 0 0 0.22
| 3277 redis-serv 127.0.0.1 28527 127.0.0.1 46648 0 0 0.27
| 22647 recordProg 127.0.0.1 46650 127.0.0.1 28527 0 0 0.21
| 3277 redis-serv 127.0.0.1 28527 127.0.0.1 46650 0 0 0.26
| [...]
```
[[monitoring.network.tcplife}]]
[[{security.audit.user]]
### ttysnoop
* <https://github.com/iovisor/bcc/blob/master/tools/ttysnoop_example.txt>
* watch a tty or pts device, and prints the same output that is
appearing on that device. It can be used to mirror the output from a shell
session, or the system console.
[[security.audit.user}]]
[[{profiling,troubleshooting,PM.low_code]]
### argdist
* <https://github.com/iovisor/bcc/blob/master/tools/argdist_example.txt>
- probe indicated functions by collecting parameter values into a
**histogram-or-frequency-count**, allowing to understand the distribution
of values a certain parameter takes, filter and print interesting parameters
**without attaching a debugger**, and obtain general execution statistics on
various functions. Ex:
```
| $ sudo ./argdist -p ${PID} \
| -c -C \
| 'p:c:malloc(size_t size):size_t:size' ← find out allocation sizes
|
| > p:c:malloc(size_t size):size_t:size
| > COUNT EVENT
| > 3 size = 16 ← 16 looks to be the most commonly
| > [01:42:34] used alloc.size
| >
| > p:c:malloc(size_t size):size_t:size
| > COUNT EVENT
| > 4 size = 16
```
[[{security.audit.user]]
### bashreadline
* <https://github.com/iovisor/bcc/blob/master/tools/bashreadline_example.txt>
* prints bash commands from all running bash shells on the system
[[security.audit.user}]]
[[{monitoring.storage,profiling.storage,troubleshooting.storage]]
### biolatency
* <https://github.com/iovisor/bcc/blob/master/tools/biolatency_example.txt>
* traces block device I/O (disk I/O), and records the distribution
of I/O latency (time), printing this as a histogram when Ctrl-C is hit
### biosnoop
* <https://github.com/iovisor/bcc/blob/master/tools/biosnoop_example.txt>
* biosnoop traces block device I/O (disk I/O), and prints a line of output
### biotop "top for disks"
* <https://github.com/iovisor/bcc/blob/master/tools/biotop_example.txt>
* lock device I/O top, biotop summarizes which processes are
performing disk I/O.
### filetop
* <https://github.com/iovisor/bcc/blob/master/tools/filetop_example.txt>
* show reads and writes by file, with process details.
### bitesize
* <https://github.com/iovisor/bcc/blob/master/tools/bitesize_example.txt>
* show I/O distribution for requested block sizes, by process name.
### cachestat
* <https://github.com/iovisor/bcc/blob/master/tools/cachestat_example.txt>
* shows hits and misses to the file system page cache.
### cachetop
* <https://github.com/iovisor/bcc/blob/master/tools/cachetop_example.txt>
* show Linux page cache hit/miss statistics including read and write hit % per
processes in a UI like top.
### dcstat
* <https://github.com/iovisor/bcc/blob/master/tools/dcstat_example.txt>
* dcstat shows directory entry cache (dcache) statistics.
### fileslower
* <https://github.com/iovisor/bcc/blob/master/tools/fileslower_example.txt>
* shows file-based synchronous reads and writes slower than a threshold.
### llcstat
* <https://github.com/iovisor/bcc/blob/master/tools/llcstat_example.txt>
* traces cache reference and cache miss events system-wide, and summarizes
them by PID and CPU.
[[{monitoring.hardware]]
### mdflush
* <https://github.com/iovisor/bcc/blob/master/tools/mdflush_example.txt>
* traces flushes at the md driver (kernel software RAID) level.
[[monitoring.hardware}]]
[[monitoring.storage}]]
[[{monitoring.eBPF,kernel.eBPF]]
### bpflist
* displays information on running eBPF programs and optionally also
prints open kprobes and uprobes.
[[monitoring.bft}]]
[[{security.audit.capable]]
### capable
* <https://github.com/iovisor/bcc/blob/master/tools/capable_example.txt>
* capable traces calls to the kernel cap_capable() function, which does
security capability checks, and prints details for each call.
[[security.audit.capable}]]
[[{monitoring.cpu,profiling.cpu]]
### cpudist
* <https://github.com/iovisor/bcc/blob/master/tools/cpudist_example.txt>
* summarizes task on-CPU time as a histogram, showing how long tasks
spent on the CPU before being descheduled.
[[monitoring.cpu}]]
[[{monitoring.cpu,profiling.cpu]]
### cpuunclaimed
* <https://github.com/iovisor/bcc/blob/master/tools/cpuunclaimed_example.txt>
* samples the length of the CPU run queues and determine when there are
idle CPUs, yet queued threads waiting their turn.
[[monitoring.cpu}]]
[[{monitoring.kernel.locks,troubleshooting.locks]]
### criticalstat
* traces and reports occurences of atomic critical sections in the
kernel with useful stacktraces showing the origin of them.
* <https://github.com/iovisor/bcc/blob/master/tools/criticalstat_example.txt>
[[monitoring.kernel.locks}]]
[[{profiling.ddbb,profiling.ddbb.postgresql,profiling.ddbb.mysql]]
### dbslower
* <https://github.com/iovisor/bcc/blob/master/tools/dbslower_example.txt>
traces queries served by a MySQL or PostgreSQL server, and prints
those that exceed a latency (query time) threshold.
### dbstat
* <https://github.com/iovisor/bcc/blob/master/tools/dbstat_example.txt>
traces queries performed by a MySQL or PostgreSQL database process, and
displays a histogram of query latencies.
[[profiling.ddbb}]]
[[{troubleshooting.locks.deadlock]]
### *deadlock*
* <https://github.com/iovisor/bcc/blob/master/tools/deadlock_example.txt>
* This program detects potential deadlocks on a running process. The program
attaches uprobes on `pthread_mutex_lock` and `pthread_mutex_unlock` to build
a mutex wait directed graph, and then looks for a cycle in this graph.
[[troubleshooting.locks.deadlock}]]
[[{troubleshooting.memory.drsnoop,profiling.memory]]
### drsnoop
* <https://github.com/iovisor/bcc/blob/master/tools/drsnoop_example.txt>
While tracing, the processes alloc pages, due to insufficient memory available
in the system, direct reclaim events happened, which will increase the waiting
delay of the processes.
* drsnoop traces the direct reclaim system-wide, and prints various details.
[[troubleshooting.memory.drsnoop}]]
[[{monitoring.jobs,security.audit.process]]
### execsnoop
* <https://github.com/iovisor/bcc/blob/master/tools/execsnoop_example.txt>
* Traces new process
[[}]]
[[{monitoring.network.gethostlatency]]
### gethostlatency
* <https://github.com/iovisor/bcc/blob/master/tools/gethostlatency_example.txt>
* traces host name lookup calls.
[[monitoring.network.gethostlatency}]]
[[{monitoring.hardware,profiling.hardware]]
### hardirqs
* <https://github.com/iovisor/bcc/blob/master/tools/hardirqs_example.txt>
* traces hard interrupts (irqs), and stores timing statistics in-kernel for efficiency.
[[monitoring.hardware}]]
[[{troubleshooting.memory.memleak,monitoring.memory,profiling.memory,]]
### memleak
* <https://github.com/iovisor/bcc/blob/master/tools/memleak_example.txt>
* trace and match memory allocation and deallocation requests, and
collects call stacks for each allocation.
* memleak can then print a summary of which call stacks performed allocations
that weren't subsequently freed.
[[troubleshooting.memory.memleak}]]
[[{profiling.locks,troubleshooting.locks]]
### offcputime, offcpu..., ...
* <https://github.com/iovisor/bcc/blob/master/tools/offcputime_example.txt>
shows stack traces that were blocked, and the total duration they
were blocked.
[[profiling.locks}]]
[[{troubleshooting.memory,monitoring.memory]]
### oomkill
* <https://github.com/iovisor/bcc/blob/master/tools/oomkill_example.txt>
* simple program that traces the Linux out-of-memory (OOM) killer,
and shows basic details on one line per OOM kill.
[[troubleshooting.memory}]]
[[{profiling.locks,troubleshooting.locks]]
### wakeuptime
* <https://github.com/iovisor/bcc/blob/master/tools/wakeuptime_example.txt>
* measures when threads block, and shows the stack traces for the
threads that performed the wakeup, along with the process names of the waker
and target processes, and the total blocked time.
[[profiling.locks}]]
[[{ profiling.java, profiling.python, profiling.nodejs]]
[[ monitoring.java,monitoring.python,monitoring.nodejs]]
## Python,Java,NodeJS
### ugc
* <https://github.com/iovisor/bcc/blob/master/tools/lib/ugc_example.txt>
* traces garbage collection events in high-level languages, including Java,
Python, Ruby, and Node.
### ucalls
* <https://github.com/iovisor/bcc/blob/master/tools/lib/ucalls_example.txt>
* ucalls summarizes method calls in various high-level languages, including Java,
Perl, PHP, Python, Ruby, Tcl, and Linux system calls.
### uflow
* <https://github.com/iovisor/bcc/blob/master/tools/lib/uflow_example.txt>
* uflow traces method entry and exit events and prints a visual flow graph that
shows how methods are entered and exited, similar to a tracing debugger with
breakpoints. This can be useful for understanding program flow in high-level
languages such as Java, Perl, PHP, Python, Ruby, and Tcl which provide USDT
probes for method invocations.
### uobjnew
* <https://github.com/iovisor/bcc/blob/master/tools/lib/uobjnew_example.txt>
* summarizes new object allocation events and prints out statistics on
which object type has been allocated frequently, and how many bytes of that
type have been allocated. This helps diagnose common allocation paths, which
can in turn cause heavy garbage collection.
### ustat
* <https://github.com/iovisor/bcc/blob/master/tools/lib/ustat_example.txt>
* ustat is a "top"-like tool for monitoring events in high-level languages. It
prints statistics about garbage collections, method calls, object allocations,
and various other events for every process that it recognizes with a Java,
Node, Perl, PHP, Python, Ruby, and Tcl runtime.
### uthreads
* <https://github.com/iovisor/bcc/blob/master/tools/lib/uthreads_example.txt>
* traces thread creation events in Java or raw (C) pthreads, and prints
details about the newly created thread. For Java threads, the thread name is
printed; for pthreads, the thread's start function is printed, if there is
symbol information to resolve it.
### TODO:
* filelife
* fileslower
* vfscount
* vfsstat
* dcstat, ...
[[}]]
[[monitoring.jobs}]]