Documentation Grafana Alloy Reference Components pyroscope pyroscope.ebpf

pyroscope.ebpf

pyroscope.ebpf configures an eBPF profiling job for the current host. The collected performance profiles are forwarded to the list of receivers passed in forward_to.

The pyroscope.ebpf component embeds the grafana/opentelemetry-ebpf-profiler which is a fork of open-telemetry/opentelemetry-ebpf-profiler.

You can specify multiple pyroscope.ebpf components by giving them different labels, however it’s not recommended as it can lead to additional memory and CPU usage.

Supported languages

• Native code (C/C++, Rust, Zig, Go, etc. without debug symbols on host)
• Broad set of HLLs (Hotspot JVM, Python, Ruby, PHP, Node.JS, V8, Perl).

Usage

alloy Copy

pyroscope.ebpf "<LABEL>" {
  targets    = <TARGET_LIST>
  forward_to = <RECEIVER_LIST>
}

Arguments

The component configures and starts a new eBPF profiling job to collect performance profiles from the current host.

You can use the following arguments with pyroscope.ebpf:

Only the forward_to and targets fields are required. Omitted fields take their default values.

Several arguments are marked as “Deprecated (no-op)”. These arguments were previously used for configuring various cache sizes and behaviors, but they no longer have any effect. They are kept for backward compatibility but will be removed in a future release. It is recommended to remove these arguments from your configuration.

Blocks

The pyroscope.ebpf component doesn’t support any blocks. You can configure this component with arguments.

Exported fields

pyroscope.ebpf doesn’t export any fields that can be referenced by other components.

Component health

pyroscope.ebpf is only reported as unhealthy if given an invalid configuration.

Debug information

• elf_cache per build id and per same file symbol tables and their sizes in symbols count.
• pid_cache per process elf symbol tables and their sizes in symbols count.
• targets currently tracked active targets.

Debug metrics

• pyroscope_ebpf_active_targets (gauge): Number of active targets the component tracks.
• pyroscope_ebpf_pprofs_total (counter): Number of pprof profiles collected by the eBPF component.
• pyroscope_ebpf_profiling_sessions_failing_total (counter): Number of profiling sessions failed.
• pyroscope_ebpf_profiling_sessions_total (counter): Number of profiling sessions completed.
• pyroscope_fanout_latency (histogram): Write latency for sending to direct and indirect components.

Profile collecting behavior

The pyroscope.ebpf component collects stack traces associated with a process running on the current host. You can use the sample_rate argument to define the number of stack traces collected per second. The default is 97.

The following labels are automatically injected into the collected profiles if you haven’t defined them. These labels can help you pin down a profiling target.

Targets

One of the following special labels must be included in each target of targets and the label must correspond to the container or process that is profiled:

• __container_id__: The container ID.
• __meta_docker_container_id: The ID of the Docker container.
• __meta_kubernetes_pod_container_id: The ID of the Kubernetes Pod container.
• __process_pid__ : The process ID.

Each process is then associated with a specified target from the targets list, determined by a container ID or process PID.

If a process’s container ID matches a target’s container ID label, the stack traces are aggregated per target based on the container ID. If a process’s PID matches a target’s process PID label, the stack traces are aggregated per target based on the process PID. Otherwise the process isn’t profiled.

Service name

The special label service_name is required and must always be present. If it’s not specified, it’s attempted to be inferred from multiple sources:

• __meta_docker_container_name
• __meta_kubernetes_namespace and __meta_kubernetes_pod_container_name
• __meta_kubernetes_pod_annotation_pyroscope_io_service_name which is a pyroscope.io/service_name Pod annotation.

If service_name isn’t specified and couldn’t be inferred, it’s set to unspecified.

Troubleshoot unknown symbols

Symbols are extracted from various sources, including:

• The .gopclntab section in Go language ELF files.
• The .symtab and .dynsym sections in the debug ELF file.
• The .symtab and .dynsym sections in the ELF file.

The search for debug files follows gdb algorithm. For example, if the profiler wants to find the debug file for /lib/x86_64-linux-gnu/libc.so.6 with a .gnu_debuglink set to libc.so.6.debug and a build ID 0123456789abcdef. The following paths are examined:

• /usr/lib/debug/.build-id/01/0123456789abcdef.debug
• /lib/x86_64-linux-gnu/libc.so.6.debug
• /lib/x86_64-linux-gnu/.debug/libc.so.6.debug
• /usr/lib/debug/lib/x86_64-linux-gnu/libc.so.6.debug

Deal with unknown symbols

Unknown symbols in the profiles you’ve collected indicate that the profiler couldn’t access an ELF file associated with a given address in the trace.

This can occur for several reasons:

• The process has terminated, making the ELF file inaccessible.
• The ELF file is either corrupted or not recognized as an ELF file.
• There is no corresponding ELF file entry in /proc/pid/maps for the address in the stack trace.

Address unresolved symbols

If you only see module names without corresponding function names, for example, /lib/x86_64-linux-gnu/libc.so.6, it indicates that the symbols couldn’t be mapped to their respective function names.

This can occur for several reasons:

• The binary has been stripped, leaving no .symtab, .dynsym, or .gopclntab sections in the ELF file.
• The debug file is missing or couldn’t be located.

To fix this for your binaries, ensure that they’re either not stripped or that you have separate debug files available. You can achieve this by running:

bash Copy

objcopy --only-keep-debug elf elf.debug
strip elf -o elf.stripped
objcopy --add-gnu-debuglink=elf.debug elf.stripped elf.debuglink

For system libraries, ensure that debug symbols are installed. On Ubuntu, for example, you can install debug symbols for libc by executing:

bash Copy

apt install libc6-dbg

Understand flat stack traces

If your profiles show many shallow stack traces, typically 1-2 frames deep, your binary might have been compiled without frame pointers.

To compile your code with frame pointers, include the -fno-omit-frame-pointer flag in your compiler options.

Example

Kubernetes discovery

In the following example, performance profiles are collected from Pods on the same node, discovered using discovery.kubernetes. Pod selection relies on the HOSTNAME environment variable, which is a Pod name if Alloy is used as an Alloy Helm chart. The service_name label is set to {__meta_kubernetes_namespace}/{__meta_kubernetes_pod_container_name} from Kubernetes meta labels.

alloy Copy

discovery.kubernetes "all_pods" {
  role = "pod"
  selectors {
    field = "spec.nodeName=" + sys.env("HOSTNAME")
    role = "pod"
  }
}

discovery.relabel "local_pods" {
  targets = discovery.kubernetes.all_pods.targets
  rule {
    action = "drop"
    regex = "Succeeded|Failed"
    source_labels = ["__meta_kubernetes_pod_phase"]
  }
  rule {
    action = "replace"
    regex = "(.*)@(.*)"
    replacement = "ebpf/${1}/${2}"
    separator = "@"
    source_labels = ["__meta_kubernetes_namespace", "__meta_kubernetes_pod_container_name"]
    target_label = "service_name"
  }
  rule {
    action = "labelmap"
    regex = "__meta_kubernetes_pod_label_(.+)"
  }
  rule {
    action = "replace"
    source_labels = ["__meta_kubernetes_namespace"]
    target_label = "namespace"
  }
  rule {
    action = "replace"
    source_labels = ["__meta_kubernetes_pod_name"]
    target_label = "pod"
  }
  rule {
    action = "replace"
    source_labels = ["__meta_kubernetes_pod_node_name"]
    target_label = "node"
  }
  rule {
    action = "replace"
    source_labels = ["__meta_kubernetes_pod_container_name"]
    target_label = "container"
  }
}
pyroscope.ebpf "local_pods" {
  forward_to = [ pyroscope.write.endpoint.receiver ]
  targets = discovery.relabel.local_pods.output
}

pyroscope.write "endpoint" {
  endpoint {
    url = "http://pyroscope:4040"
  }
}

Docker discovery

The following example collects performance profiles from containers discovered by discovery.docker and ignores all other profiles collected from outside any docker container. The service_name label is set to the __meta_docker_container_name label.

alloy Copy

discovery.docker "linux" {
  host = "unix:///var/run/docker.sock"
}

discovery.relabel "local_containers" {
  targets = discovery.docker.linux.targets
  rule {
    action = "replace"
    source_labels = ["__meta_docker_container_name"]
    target_label = "service_name"
  }
}

pyroscope.write "staging" {
  endpoint {
    url = "http://pyroscope:4040"
  }
}

pyroscope.ebpf "default" {
  forward_to   = [ pyroscope.write.staging.receiver ]
  targets      = discovery.relabel.local_containers.output
}

Compatible components

pyroscope.ebpf can accept arguments from the following components:

• Components that export Targets
• Components that export Pyroscope ProfilesReceiver