Skip to content

feat: add pprof endpoint option for profiling #331

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
hengyoush wants to merge 1 commit into
base: main
Choose a base branch
from
Open

feat: add pprof endpoint option for profiling #331

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
50 changes: 49 additions & 1 deletion agent/agent.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,21 +19,23 @@ import (
"kyanos/version"
"os"
"os/exec"
"net/http"
"os/signal"
"runtime"
"strings"
"sync"
"syscall"
"time"

_ "net/http/pprof"
"net/http/pprof"

"github.com/cilium/ebpf/rlimit"
gops "github.com/google/gops/agent"
)

func SetupAgent(options ac.AgentOptions) {
startGopsServer(options)
startPprofServer(options)

Comment on lines 36 to 39
Copy link

Copilot AI Mar 5, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

startPprofServer(options) is called before options.Ctx is initialized (it’s set later via signal.NotifyContext). When --pprof is enabled, startPprofServer will block on <-opts.Ctx.Done() and will panic if Ctx is nil. Move the pprof startup to after options = ValidateAndRepairOptions(options) and after options.Ctx is set, or make startPprofServer tolerate a nil context (e.g., skip shutdown hook / use Background).

Copilot uses AI. Check for mistakes.
if err := version.UpgradeDetect(); err != nil {
if errors.Is(err, version.ErrBehindLatest) {
Expand Down Expand Up @@ -252,3 +254,49 @@ func startGopsServer(opts ac.AgentOptions) {
}
}
}

func startPprofServer(opts ac.AgentOptions) {
if !opts.EnablePprof {
return
}

addr := opts.GetPprofAddr()
displayAddr := addr
if strings.HasPrefix(addr, ":") {
displayAddr = "localhost" + addr
}

mux := http.NewServeMux()
mux.HandleFunc("/debug/pprof/", pprof.Index)
mux.HandleFunc("/debug/pprof/cmdline", pprof.Cmdline)
mux.HandleFunc("/debug/pprof/profile", pprof.Profile)
mux.HandleFunc("/debug/pprof/symbol", pprof.Symbol)
mux.HandleFunc("/debug/pprof/trace", pprof.Trace)
mux.HandleFunc("/debug/pprof/{name}", pprof.Index)

server := &http.Server{
Addr: addr,
Handler: mux,
ReadHeaderTimeout: 10 * time.Second,
IdleTimeout: 120 * time.Second,
}

go func() {
common.AgentLog.Infof("Starting pprof server on http://%s/debug/pprof/", displayAddr)
common.AgentLog.Info("pprof endpoints: /debug/pprof/heap, /debug/pprof/profile, /debug/pprof/goroutine")
if err := server.ListenAndServe(); err != nil && err != http.ErrServerClosed {
common.AgentLog.Errorf("Failed to start pprof server: %v", err)
}
}()

go func() {
<-opts.Ctx.Done()
shutdownCtx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
defer cancel()
if err := server.Shutdown(shutdownCtx); err != nil {
common.AgentLog.Errorf("Failed to shutdown pprof server: %v", err)
} else {
common.AgentLog.Info("pprof server shutdown gracefully")
}
}()
}
9 changes: 9 additions & 0 deletions agent/common/options.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -50,6 +50,8 @@ type AgentOptions struct {
ConntrackCloseWaitTimeMills int
MaxAllowStuckTimeMills int
StartGopsServer bool
EnablePprof bool
PprofAddr string

FilterComm string
ProcessExecEventChannel chan *bpf.AgentProcessExecEvent
Expand All @@ -74,6 +76,13 @@ type AgentOptions struct {
FirstPacketEventMapPageNum int
}

func (o AgentOptions) GetPprofAddr() string {
if o.PprofAddr == "" {
return "localhost:6060"
}
return o.PprofAddr
}
Comment on lines +79 to +84
Copy link

Copilot AI Mar 5, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

GetPprofAddr() falls back to "localhost:6060", while the CLI flag default is 127.0.0.1:6060 and docs mention localhost:6060. Using localhost can also resolve to IPv6-only on some systems. Consider using a single constant/default (preferably 127.0.0.1:6060) across flags, docs, and this fallback to avoid surprising bind behavior.

Copilot uses AI. Check for mistakes.

func (o AgentOptions) FilterByContainer() bool {
return o.ContainerId != "" || o.ContainerName != "" || o.PodName != ""
}
Expand Down
2 changes: 2 additions & 0 deletions cmd/root.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -105,6 +105,8 @@ func init() {
rootCmd.PersistentFlags().IntVar(&options.MaxAllowStuckTimeMills, "max-allow-stuck-time-mills", 1000, "--max-allow-stuck-time-mills 100")

rootCmd.PersistentFlags().BoolVar(&options.StartGopsServer, "gops", false, "start gops server")
rootCmd.PersistentFlags().BoolVar(&options.EnablePprof, "pprof", false, "enable pprof HTTP endpoint for profiling (WARNING: exposes sensitive profiling data; do not use on untrusted networks)")
rootCmd.PersistentFlags().StringVar(&options.PprofAddr, "pprof-addr", "127.0.0.1:6060", "pprof HTTP server address (WARNING: binding to non-loopback addresses, e.g. \":6060\" or \"0.0.0.0:6060\", will expose profiling data to the network)")

rootCmd.PersistentFlags().MarkHidden("default-log-level")
rootCmd.PersistentFlags().MarkHidden("agent-log-level")
Expand Down
44 changes: 44 additions & 0 deletions docs/cn/debug-tips.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -85,6 +85,50 @@ VSCODE 直接打开项目即可，.vscode/launch.json 添加配置如下：

注意添加 `--debug-output` 参数。

## 性能分析（Profiling）

Kyanos 提供了 pprof 端点用于运行时性能分析和调试。你可以使用 `--pprof` 标志启用 pprof HTTP 服务器：

```bash
./kyanos watch --pprof
```

默认情况下，pprof 服务器监听 `localhost:6060`。你可以使用 `--pprof-addr` 标志自定义监听地址：

```bash
./kyanos watch --pprof --pprof-addr="0.0.0.0:9090"
Copy link

Copilot AI Mar 5, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

此示例使用 --pprof-addr="0.0.0.0:9090" 启动 pprof HTTP 服务器，会在没有任何认证或 TLS 的情况下监听所有网卡。结合当前 --pprof 实现，只要能访问该端口的主机就可以获取 heap/CPU profile 和 goroutine dump，这些数据中往往包含敏感内存信息。建议文档示例保持绑定在本机地址，并明确提示只有在有严格网络控制（如 SSH 隧道或防火墙）时才应使用非回环地址，否则会把 profiling 数据暴露到网络。

Copilot uses AI. Check for mistakes.
```

可用的 pprof 端点：

| 端点 | 说明 |
|------|------|
| `/debug/pprof/` | 索引页面，显示所有可用的 profile |
| `/debug/pprof/heap` | 内存堆 profile |
| `/debug/pprof/profile` | CPU profile（默认 30 秒） |
| `/debug/pprof/goroutine` | Goroutine 堆栈信息 |
| `/debug/pprof/allocs` | 内存分配 profile |
| `/debug/pprof/block` | 阻塞 profile |
| `/debug/pprof/mutex` | 锁竞争 profile |
Comment on lines +104 to +112
Copy link

Copilot AI Mar 5, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

该 markdown 表格格式不正确（每行以 || 开头而不是 |），在 GitHub 上可能无法正常渲染。建议去掉多余的前导 |，使用标准的 | ... | 表格语法。

Copilot uses AI. Check for mistakes.

使用示例：

```bash
# 采集 CPU profile
curl -o cpu.pprof http://localhost:6060/debug/pprof/profile?seconds=30
go tool pprof cpu.pprof

# 查看堆内存 profile
go tool pprof http://localhost:6060/debug/pprof/heap

# 查看 goroutine 堆栈
curl http://localhost:6060/debug/pprof/goroutine?debug=1
```

> [!TIP]
>
> pprof 端点对于诊断 Kyanos 自身的性能问题、内存泄漏或 goroutine 泄漏非常有用。

## 源码结构

```
Expand Down
44 changes: 44 additions & 0 deletions docs/debug-tips.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -94,6 +94,50 @@ Make sure to add the `--debug-output` parameter.
Protocol inference code can be debugged by printing logs using `bpf_printk`,
refer to: https://nakryiko.com/posts/bpf-tips-printk/.

## Profiling

Kyanos provides pprof endpoints for runtime profiling and debugging. You can enable the pprof HTTP server using the `--pprof` flag:

```bash
./kyanos watch --pprof
```

By default, the pprof server listens on `localhost:6060`. You can customize the address using the `--pprof-addr` flag:

```bash
./kyanos watch --pprof --pprof-addr="0.0.0.0:9090"
Copy link

Copilot AI Mar 5, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This example runs the pprof HTTP server with --pprof-addr="0.0.0.0:9090", which binds the profiling endpoint to all network interfaces with no authentication or TLS. In combination with the current --pprof implementation, any host that can reach this port can download heap/CPU profiles and goroutine dumps, which commonly contain sensitive in‑memory data. Consider keeping the example bound to localhost and explicitly warning that non-loopback addresses should only be used behind strong network controls (e.g. SSH tunnel or firewall) because they expose profiling data to the network.

Copilot uses AI. Check for mistakes.
```

Available pprof endpoints:

| Endpoint | Description |
|----------|-------------|
| `/debug/pprof/` | Index page with all available profiles |
| `/debug/pprof/heap` | Memory heap profile |
| `/debug/pprof/profile` | CPU profile (30 seconds by default) |
| `/debug/pprof/goroutine` | Goroutine dump |
Comment on lines +113 to +118
Copy link

Copilot AI Mar 5, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The markdown table is malformed (each row starts with || instead of |), which won’t render correctly on GitHub. Remove the extra leading | so the table uses standard | ... | syntax.

Copilot uses AI. Check for mistakes.
| `/debug/pprof/allocs` | Allocation profile |
| `/debug/pprof/block` | Block profile |
| `/debug/pprof/mutex` | Mutex contention profile |

Example usage:

```bash
# Capture CPU profile
curl -o cpu.pprof http://localhost:6060/debug/pprof/profile?seconds=30
go tool pprof cpu.pprof

# View heap profile
go tool pprof http://localhost:6060/debug/pprof/heap

# View goroutine dump
curl http://localhost:6060/debug/pprof/goroutine?debug=1
```

> [!TIP]
>
> The pprof endpoint is useful for diagnosing performance issues, memory leaks, or goroutine leaks in Kyanos itself.

## Source Code Structure

```
Expand Down
Loading