extend ecapture to profile CPU usage?

[vincentmli]

Hi @cfc4n

I was searching if there is eBPF kprobe/uprobe tool to profile CPU usage like linux perf tool does perf record, perf report, or perf top. I found bcc tool https://github.com/iovisor/bcc/blob/master/tools/offcputime.py does exactly that, but bcc is heavy package and not installed for blackbox environment. I like the portability of ecapture, could ecapture be extended to have bcc offcputime like feature to collect cpu kernel/user stack for CPU usage profiling and the collected data could be used to create flame graph, I think this would extend ecapture to even bigger user space, I am new to eBPF kprobe/uprobe, but I am learning it everyday, if you don't have time to do it, I could learn to do it but need guidance :-)

Vincent

[cfc4n]

My idea is that keep eCapture as monolithic as possible. just TLS/SSL. Sometimes I want to remove bash/mysqld modules.

But I think this idea of yours should be needed by a lot of people.

So, I think :

1. create a new project is better merge into eCapture.

2. In other words, using eCapture as a framework, you can fork yourself and customize the functions.

which Option would you choose ?

[vincentmli]

I agree with you, like the Unix philosophy, do one thing and do one thing well. this may not happen soon though since I am not sure if i would have the time because I am still learning eBPF tracing and ecapture framework, and at the same time contributing to cilium network projects, if it is easier for you to start it up, I could help contributing :-)

[cfc4n]

I have a new plan about ecapture , It's save packet to pcappng type file. be opened by Wireshark .

Now , I am developing it. maybe will be completed in a month.

You can give more detial about your idea.

[vincentmli]

as a simple start, I think it is just matter of porting https://github.com/iovisor/bcc/blob/master/tools/offcputime.py to ecapture framework based golang binary app that is portable to run in as many environment as possible, I tried BCC offcputime.py in my Ubuntu 22.04, it failed to run after I install the ubuntu package, that is not good user experience! so I prefer golang based single binary that include every dependencies compiled in the binary already and runs everywhere

apt-get install bpfcc-tools

offcputime-bpfcc -U
In file included from <built-in>:2:
In file included from /virtual/include/bcc/bpf.h:12:
In file included from include/linux/types.h:6:
In file included from include/uapi/linux/types.h:14:
In file included from include/uapi/linux/posix_types.h:5:
In file included from include/linux/stddef.h:5:
In file included from include/uapi/linux/stddef.h:2:
In file included from include/linux/compiler_types.h:80:
include/linux/compiler-clang.h:41:9: warning: '__HAVE_BUILTIN_BSWAP32__' macro redefined [-Wmacro-redefined]
#define __HAVE_BUILTIN_BSWAP32__
        ^
<command line>:4:9: note: previous definition is here
#define __HAVE_BUILTIN_BSWAP32__ 1
        ^
In file included from <built-in>:2:
In file included from /virtual/include/bcc/bpf.h:12:
In file included from include/linux/types.h:6:
In file included from include/uapi/linux/types.h:14:
In file included from include/uapi/linux/posix_types.h:5:
In file included from include/linux/stddef.h:5:
In file included from include/uapi/linux/stddef.h:2:
In file included from include/linux/compiler_types.h:80:
include/linux/compiler-clang.h:42:9: warning: '__HAVE_BUILTIN_BSWAP64__' macro redefined [-Wmacro-redefined]
#define __HAVE_BUILTIN_BSWAP64__
        ^
<command line>:5:9: note: previous definition is here
#define __HAVE_BUILTIN_BSWAP64__ 1
        ^
In file included from <built-in>:2:
In file included from /virtual/include/bcc/bpf.h:12:
In file included from include/linux/types.h:6:
In file included from include/uapi/linux/types.h:14:
In file included from include/uapi/linux/posix_types.h:5:
In file included from include/linux/stddef.h:5:
In file included from include/uapi/linux/stddef.h:2:
In file included from include/linux/compiler_types.h:80:
include/linux/compiler-clang.h:43:9: warning: '__HAVE_BUILTIN_BSWAP16__' macro redefined [-Wmacro-redefined]
#define __HAVE_BUILTIN_BSWAP16__
        ^
<command line>:3:9: note: previous definition is here
#define __HAVE_BUILTIN_BSWAP16__ 1
        ^
3 warnings generated.
cannot attach kprobe, probe entry may not exist
Traceback (most recent call last):
  File "/usr/sbin/offcputime-bpfcc", line 234, in <module>
    b.attach_kprobe(event="finish_task_switch", fn_name="oncpu")
  File "/usr/lib/python3/dist-packages/bcc/__init__.py", line 683, in attach_kprobe
    raise Exception("Failed to attach BPF program %s to kprobe %s" %
Exception: Failed to attach BPF program b'oncpu' to kprobe b'finish_task_switch'

[vincentmli]

it looks kernel people has the same idea but with perf record, it is still in RFC stage though, and I think perf tool also relies on linux header files to be installed on target machine, so I still think golang based single static binary brings better user experience.
[RFC 0/6] perf record: Implement off-cpu profiling with BPF (v3)

[vincentmli]

hm, I found out that some BCC tools are actually ported to libbpf https://github.com/iovisor/bcc/tree/master/libbpf-tools that are completely written in C, the ported tool including offcputime that I compiled on Centos8, and can also run on Ubuntu 22.04, after I ran offcputime, I think it is not actually what I wanted, [RFC 0/6] perf record: Implement off-cpu profiling with BPF (v3) is closer to what I want. not sure if it is easy to implement based on ecapture framework though

[cfc4n]

very simple.

just 5 files.

1. cli/cmd/new_cmd.go
2. kern/new_cmd_kern.c
3. user/config_new_cmd.go
4. user/event_new_cmd.go
5. user/probe_new_cmd.go

and see https://ecapture.cc/zh/develop/ for more detail.

[vincentmli]

fyi, I found this profile from iovisor/bcc#3782 that is libbpf based. now I have some basic idea about how eBPF tracing works after reading through BPF mailing list archive for a few weeks :). my impression is libbpf seems got more portability/feature support from mainline core kernel developers directly, not sure if cilium/ebpf library matches the feature/function of libbpf, especially in regard to kernel tracing area. for example, [PATCH v5 bpf-next 2/5] libbpf: support function name-based attach uprobes, my understanding is that with this feature in libbpf, you don't need to write your own symbol resolution, libbpf automatically handle that for you, right ? @cfc4n

[cfc4n]

libbpf does not contain this feature about symbol resolution. It just search the offset of symbol address. You must do it by yourself.

cilium also can search offset when symbol name was set.

[vincentmli]

in the libbpf example of https://lore.kernel.org/bpf/[email protected]/

SEC("uprobe//usr/lib64/libc.so.6:malloc"), this is to say for uprobe, find the symbols of /usr/lib64/libc.so.6, and get the offset address of malloc, correct? and cilium/ebpf can do the same, right? like ecapture does. I might be confused with symbol resolution of kprobe in libbpf and libbpf will do symbol resolution for kprobe kernel functions in /proc/kallsyms, let me if I am wrong.

[cfc4n]

yep, cilium/ebpf will read symbols from ELF file ,and save as a map for uprobe hook, at https://github.com/cilium/ebpf/blob/5bd5446279c4b526a4282d5bb0bafafb2e607e7b/link/uprobe.go#L120-L164

for _, s := range syms {
		if elf.ST_TYPE(s.Info) != elf.STT_FUNC {
			// Symbol not associated with a function or other executable code.
			continue
		}

		address := s.Value

		// Loop over ELF segments.
		for _, prog := range f.Progs {
			// Skip uninteresting segments.
			if prog.Type != elf.PT_LOAD || (prog.Flags&elf.PF_X) == 0 {
				continue
			}

			if prog.Vaddr <= s.Value && s.Value < (prog.Vaddr+prog.Memsz) {
				// If the symbol value is contained in the segment, calculate
				// the symbol offset.
				//
				// fn symbol offset = fn symbol VA - .text VA + .text offset
				//
				// stackoverflow.com/a/40249502
				address = s.Value - prog.Vaddr + prog.Off
				break
			}
		}

		ex.addresses[s.Name] = address
	}

[cfc4n]

yep, cilium/ebpf will read symbols from ELF file ,and save as a map for uprobe hook, at https://github.com/cilium/ebpf/blob/5bd5446279c4b526a4282d5bb0bafafb2e607e7b/link/uprobe.go#L120-L164

for _, s := range syms {
		if elf.ST_TYPE(s.Info) != elf.STT_FUNC {
			// Symbol not associated with a function or other executable code.
			continue
		}

		address := s.Value

		// Loop over ELF segments.
		for _, prog := range f.Progs {
			// Skip uninteresting segments.
			if prog.Type != elf.PT_LOAD || (prog.Flags&elf.PF_X) == 0 {
				continue
			}

			if prog.Vaddr <= s.Value && s.Value < (prog.Vaddr+prog.Memsz) {
				// If the symbol value is contained in the segment, calculate
				// the symbol offset.
				//
				// fn symbol offset = fn symbol VA - .text VA + .text offset
				//
				// stackoverflow.com/a/40249502
				address = s.Value - prog.Vaddr + prog.Off
				break
			}
		}

		ex.addresses[s.Name] = address
	}

and used at link.Uprobe() called. in link/uprbe.go line 275

https://github.com/cilium/ebpf/blob/5bd5446279c4b526a4282d5bb0bafafb2e607e7b/link/uprobe.go#L275

	offset, err := ex.address(symbol, opts)