Diagnosing and Resolving Go Memory Leak with pprof and Prometheus

On a sunny day the author received a high memory‑usage alert from a Go service that had been gradually migrated from Python. The container’s RSS grew until it hit the memory limit and the process restarted.

To investigate, the go tool pprof utility was used against the /debug/pprof/allocs endpoint:

go tool pprof -source_path=/path/to/gopath -inuse_space https://target.service.url/debug/pprof/allocs

The top 10 command showed that the function github.com/beorn7/perks/quantile.newStream consumed ~75% of the total memory:

(pprof) top 10 Showing nodes accounting for 145.07MB, 92.64% of 156.59MB total ... 117.36MB 74.95% github.com/beorn7/perks/quantile.newStream (inline)

Inspecting the source of newStream revealed a pre‑allocated slice of 500 Sample structs (each 24 bytes), i.e., ~12 KB per call.

type Sample struct { Value float64 `json:",string"` Width float64 `json:",string"` Delta float64 `json:",string"` }

Further analysis of the Prometheus metric vector showed that a label array was built for every request, combining operation type, host, database, table name, error flag, SQL state, endpoint, and start‑point. The combinatorial explosion of the startpoint label caused a new metric to be created for each distinct request URL, leading to unbounded memory growth.

The relevant code for label construction:

label := append([]string{getOperation(db), s.host, s.database, tableName, hasErr, sqlState}, metrics.InjectTagValue(collector.MetricsTitle, db.Statement.Context, attachment)...)

Prometheus creates a new metric when the label set is unseen, as shown in the metric vector implementation:

func (m *MetricVec) GetMetricWithLabelValues(lvs ...string) (Metric, error) { h, err := m.hashLabelValues(lvs) if err != nil { return nil, err } return m.metricMap.getOrCreateMetricWithLabelValues(h, lvs, m.curry), nil } func (m *MetricVec) hashLabelValues(vals []string) (uint64, error) { // hash each label value }

The fix was to disable the START_POINT environment variable, which stops the endpoint address from being added to the metric labels. After redeploying with this change, RSS stabilized and the memory leak disappeared.

Finally, the article lists common Go memory‑leak patterns such as goroutine leaks, slice/string sharing, pointer slices, array copying, and timer misuse, reminding developers to monitor and profile their applications regularly.