The OpenTelemetry eBPF project develops components that collect and analyze telemetry from the operating system, cloud, and container orchestrators. Its initial focus is on collecting network data to enable users to gain insight into their distributed applications.
The kernel collector gathers low level telemetry straight from the Linux kernel using eBPF. It does so with negligible compute and network overheads. The kubernetes collector and cloud collector gather workload metadata.
This telemetry is then sent to the reducer, which enriches and aggregates it. The reducer outputs metrics to the OpenTelemetry collector.
Before building, make sure that all submodules are checked-out:
git submodule update --init --recursive
There's a docker build image provided with all dependencies pre-installed, ready to build the collectors.
Building the collectors images is as simple as running the build image within docker with the following setup:
docker run \
-it --rm \
--mount "type=bind,source=/var/run/docker.sock,destination=/var/run/docker.sock" \
--mount "type=bind,source=$(git rev-parse --show-toplevel),destination=/root/src,readonly" \
--env EBPF_NET_SRC=/root/src \
--env EBPF_NET_OUT_DIR=/root/out \
--workdir=/root/out \
build-env \
../build.sh docker
The resulting docker image will be placed in the host's docker daemon under the
name kernel-collector
.
The images can also be automatically pushed to a docker registry after they're built.
By default, they're pushed to a local docker registry at localhost:5000
. The registry
can be changed by setting the environment variable EBPF_NET_DOCKER_REGISTRY
in the
build image, as so:
docker run \
-it --rm \
--mount "type=bind,source=/var/run/docker.sock,destination=/var/run/docker.sock" \
--mount "type=bind,source=$(git rev-parse --show-toplevel),destination=/root/src,readonly" \
--env EBPF_NET_SRC=/root/src \
--env EBPF_NET_OUT_DIR=/root/out \
--env EBPF_NET_DOCKER_REGISTRY="localhost:5000" \
--workdir=/root/out \
build-env \
../build.sh docker-registry
The source code for the build image as well as instructions on how to build it can be found in its repo at github.com/Flowmill/flowmill-build-env.
Running the kernel collector should be as easy as running a docker image:
docker run -it --rm \
--env EBPF_NET_INTAKE_PORT="${EBPF_NET_INTAKE_PORT}" \
--env EBPF_NET_INTAKE_HOST="${EBPF_NET_INTAKE_HOST}" \
--privileged \
--pid host \
--network host \
--log-console \
--volume /var/run/docker.sock:/var/run/docker.sock \
--volume /sys/fs/cgroup:/hostfs/sys/fs/cgroup \
--volume /etc:/hostfs/etc \
--volume /var/cache:/hostfs/cache \
--volume /usr/src:/hostfs/usr/src \
--volume /lib/modules:/hostfs/lib/modules \
kernel-collector \
--log-console
Environment variables:
EBPF_NET_INTAKE_HOST
: this is the hostname or IP address of the intake serverEBPF_NET_INTAKE_PORT
: this is the port of the intake server
Volumes:
/var/run/docker.sock
: enables the collector to talk to the local Docker daemon/sys/fs/cgroup
: allows the collector to read cgroup information/etc
: allows the collector to read package manager settings in order to fetch kernel headers in case they're not pre-installed on the host (necessary for eBPF - optional if pre-installed kernel headers are available on the host)/var/cache
: cache fetched kernel headers on the host (optional)/usr/src
//lib/modules
: allows the collector to use kernel headers pre-installed on the host(necessary for eBPF)
Docker settings:
The collector needs privileged access since it uses the eBPF mechanism from the
Linux kernel, therefore these settings need to be passed to docker: --privileged
,
--pid host
and --network host
.
Kernel collector can alternatively send telemetry to OpenTelemetry Collector (otel-col) in the form of Log entries.
A minimal config is included in dev/otel/otel-config.yaml
, and it is possible to run
the standard otel distribution docker image with:
docker run -v $PWD/dev/otel/otel-config.yaml:/etc/otel/config.yaml -p 8000:4318 otel/opentelemetry-collector
A few changes need to be made to otel-col's config file. Please refer to otel-col's documentation for details on how to run the collector.
First you need to set up an HTTP endpoint for an OTLP receiver. The example
below binds the receiver to all interfaces (0.0.0.0
) on port 8000
. For more
information, refer to otel-col's
documentation:
receivers:
otlp:
protocols:
http:
endpoint: 0.0.0.0:8000
If you need to enable TLS on the endpoint, check the documentation.
Then make sure the Log Service is also enabled and connected to the OTLP HTTP receiver. For more information, refer to otel-col's documentation:
service:
pipelines:
logs:
receivers: [otlp]
# processors: # TODO: add processors here
# exporters: # TODO: add exporters here
By making sure the Log Service is enabled in otel-col and receiving HTTP
requests in OTLP format, now kernel collector is able to send telemetry to
otel-col on port 8000
.
For more information on the OTLP receiver, refer to otel-col's documentation.
The kernel collector needs to know a few things in order to connect to
otel-col's receiver as its intake. The difference between connecting to the
standard intake vs connecting to otel-col's receiver is the intake encoding.
For otel-col's receiver the encoding must be set to otlp_log
.
Intake settings are controlled by environment variables set on kernel
collector's container (e.g.: can be set with docker
's --env command line
argument). Below is a list of settings along with the name of the environment
variable and suggested values for a proof-of-concept (note that these are already
present in the docker run
command below):
export EBPF_NET_INTAKE_HOST=127.0.0.1 # host
export EBPF_NET_INTAKE_PORT=8000 # port
export EBPF_NET_INTAKE_ENCODER=otlp_log # encoder
Here's an example:
docker run -it --rm \
--env EBPF_NET_INTAKE_HOST="127.0.0.1" \
--env EBPF_NET_INTAKE_PORT="8000" \
--env EBPF_NET_INTAKE_ENCODER="otlp_log" \
--privileged \
--pid host \
--network host \
--volume /var/run/docker.sock:/var/run/docker.sock \
--volume /sys/fs/cgroup:/hostfs/sys/fs/cgroup \
--volume /etc:/hostfs/etc \
--volume /var/cache:/hostfs/cache \
--volume /usr/src:/hostfs/usr/src \
--volume /lib/modules:/hostfs/lib/modules \
kernel-collector \
--log-console
Maintainers (@open-telemetry/ebpf-maintainers):
- Borko Jandras, Splunk
- Jim Wilson, Splunk
- Jonathan Perry, Splunk
Learn more about roles in the community repository.