Section: archive
Kind: Archive · commissioned
Reading time: 8 min

Continuous Profiling with eBPF in Kubernetes

11 November 2026

Running a profiler on every node in a Kubernetes cluster means putting a privileged agent on every machine that serves production traffic. The agent has to be root, has to see the host's process table, and loads kernel-level instrumentation. Before anyone asks what the flame graphs will show, the first question should be what the rollout will cost in privilege, overhead and exposed metadata. The guardrails below draw on Grafana's documentation for its Alloy agent and on a 2026 guide from Edilec on running continuous profiling in production.

Start with the right workload and profile type

The Edilec guide is blunt about where to begin: with services that have recurring CPU cost, unexplained latency, or expensive capacity. That rules out the instinct to turn profiling on everywhere at once so the data is "there when we need it." A service that burns a predictable share of a large node pool every day is a far better first candidate than a small internal tool nobody has complained about. The first target should be one where a finding translates into something concrete, such as fewer nodes, a smaller bill or a fixed latency regression.

The same guide recommends being deliberate about which kind of profile you collect. CPU sampling is broadly useful and works across many workloads, which is why it is the sensible default. Wall-clock, off-CPU, allocation, mutex and language-runtime profiles depend on different mechanisms and on what a given tool supports. A team chasing latency that does not show up as CPU time, for example a service spending most of its request waiting, may find that a CPU profile looks unremarkable while the problem sits somewhere else entirely.

That distinction should shape expectations before the first pilot. If the question is "why is this service slow while the CPUs are idle," a CPU-only setup cannot answer it, and the team should know that going in rather than discover it after weeks of collection. Language runtimes add another wrinkle. Profiles tied to a particular runtime depend on tool support for that runtime, so a mixed fleet will not get uniform coverage just because the agent is uniform.

Pick one service, name the question you want answered, and choose the profile type that can answer it. Starting with CPU sampling and widening only when it falls short keeps the privilege footprint, and the data volume, as small as the problem allows.

Measure overhead under production-like conditions

Overhead is the number everyone wants up front, and the Edilec guide advises against treating it as a fixed property of the tool. It should be measured as a budget, not a promise. Vendor claims and community anecdotes describe somebody else's kernel, somebody else's traffic and somebody else's code. Your budget is whatever your service-level objectives can absorb, and the only way to know whether the profiler fits inside it is to test.

The method the guide describes is a comparison against a no-profiler baseline. Run the workload without the profiler, record latency, CPU use and whatever else your service-level indicators track, then run it again with the profiler attached and compare. The comparison only means something if the conditions match what you plan to run in production: the same kernel, the same node shape, the same workload, the same traffic and the same symbol path.

Each of those variables can quietly invalidate a result. Kernel versions differ in what they support and how they behave, so a benchmark on a developer-friendly image may say little about the kernel on your production nodes. A smaller or differently configured node shape changes how much headroom the agent competes for. Synthetic traffic that does not resemble the real request mix can hide the cost that appears under real load. The symbol path matters too, since resolving addresses to function names is part of the work the profiler does. Benchmarking with symbols readily available and then deploying into an environment where resolution is slow or incomplete tests the wrong thing.

A practical caveat for US teams running across several regions or several cloud providers: if your fleet mixes node shapes or kernel versions, one clean benchmark does not cover them all. Repeat the baseline comparison for each distinct combination you intend to run, or at least for the most constrained one. A budget that holds on your smallest, busiest node type is a much stronger basis for a rollout decision than an average across the fleet.

Write the budget down before testing, with the specific thresholds that would make you stop. Without that, a measured overhead becomes a number people argue about instead of a decision rule.

Run eBPF profilers with the minimum safe privilege

Grafana's documentation for the `pyroscope.ebpf` component in Alloy sets out the operating requirements plainly. The component embeds the `grafana/opentelemetry-ebpf-profiler`, which is a fork of `open-telemetry/opentelemetry-ebpf-profiler`. Alloy must run as root and inside the host PID namespace. Both requirements follow from the job: the agent needs to see processes across the node, not just inside its own container.

On Kubernetes, the Alloy documentation names the simplest route as setting `securityContext.privileged: true`. It is also the broadest. A fully privileged container is a large grant to hand a telemetry agent, and many clusters enforce admission policies that will reject it outright. The documentation therefore also lists Linux capabilities for running without full privilege: `BPF`, `PERFMON`, `SYS_PTRACE`, `CHECKPOINT_RESTORE`, `SYS_RESOURCE`, `DAC_READ_SEARCH` and `SYSLOG`.

Granting that explicit set instead of blanket privilege narrows what the container can do, and it gives a security review something specific to evaluate. Reviewers can go through the list item by item rather than reacting to the word "privileged." It is still a substantial set, and the agent still runs as root in the host PID namespace, so the capability route reduces the grant without making the pod low-risk.

Two further details trip up deployments. The documentation says `/sys/kernel/tracing` should be mounted read-only from the host so the tracer can attach tracepoints. If that mount is missing from the pod spec, the symptom may be a profiler that starts but collects nothing useful, which is easy to misread as a configuration problem elsewhere. The profiler also requires file system storage at `/tmp/symb-cache` for symbol cache data. On nodes or pods with a read-only root filesystem, that path needs a writable volume, or the agent will not have the scratch space it expects.

Treat these as part of the deployment manifest, reviewed like any other host-level access: root, host PID namespace, a defined capability set, one read-only host mount and one writable cache path. If any of them surprises the platform team during review, that is a conversation to have before the DaemonSet reaches a production node.

Treat profiling data as sensitive host metadata

Profiles feel harmless because they are "just stack traces," but the Edilec guide cautions that they can expose function names, file paths, package names, build IDs, process arguments in some implementations, and architecture details. Taken together, that describes what software runs on a host, how it was built and where it lives on disk. Process arguments, where an implementation captures them, are the most obvious worry, since command lines sometimes carry values their authors never meant to share. Build IDs and package names tell an observer exactly which versions are deployed, which is useful to an attacker looking for a known weakness.

That makes the profile backend a store of sensitive host metadata, and it should be governed that way. The Edilec guide also says the Linux BPF documentation is the authority for kernel mechanisms, and that product abstractions do not remove the need to understand what is loaded on a host. In practice, that means the security team should be able to say what the agent loads into the kernel, not just what the vendor's overview says.

The main risks and the controls that match them:

Exposed function names, file paths and package names: restrict who can query profile data to the teams that own the services, and treat the backend like any other store of internal telemetry.
Process arguments, in implementations that capture them: check what your chosen setup collects during the lab stage, before any production node is involved.
Build IDs and architecture details: assume they reveal your deployed versions and hardware, and apply the same access limits.
Broad host privilege on the agent: grant the listed capabilities instead of `privileged: true` where your policies allow it, and mount `/sys/kernel/tracing` read-only.
Unreviewed kernel-level behavior: have the team read the Linux BPF documentation for the mechanisms involved and record what the agent loads on a host.
Unbounded deployment: tie each rollout stage to a named owner and an agreed set of stop conditions.

Retention and access are decisions to make early. Profile data that seems unremarkable in a pilot can accumulate into a detailed map of the fleet, and it is much easier to set narrow access at the start than to tighten it after dozens of engineers have come to depend on it.

Roll out gradually and keep an exit path

The Edilec guide lays out the order of operations: a lab first, then non-production nodes, then a small production canary, then one workload cohort, and finally bounded fleet waves. Each step answers a different question. The lab establishes that the agent starts, mounts what it needs and produces readable profiles. Non-production nodes show how it behaves on real cluster configuration and admission policy. The canary tests the overhead budget against live traffic, and the single cohort shows whether the results hold across a family of related services before the blast radius grows.

The final stage is where discipline tends to slip. "Bounded fleet waves" means each wave has a defined size and a pause long enough to compare against the baseline you built earlier. Rolling out to every node in one change turns a contained experiment into a fleet-wide dependency on a privileged agent.

Because the agent runs as root in the host PID namespace and loads kernel-level instrumentation, the exit path deserves as much planning as the entry. Before the canary, decide how the agent is removed from a node, who can trigger that, and what evidence confirms it is gone. Rolling back a DaemonSet is the easy part. The harder question is whether anyone on call knows what the agent loaded on the host and can verify the node is back to its prior state, which is exactly where the guide's point about understanding what is loaded earns its keep.

Write the rollback runbook while the lab results are fresh, and rehearse it on non-production nodes. If removing the profiler requires heroics, the rollout is moving faster than the team's ability to control it.