Operations 10 min read

Promoter: Rendering AlertManager Graphs for DingTalk Notifications Using Go

The article introduces Promoter, a Go‑based webhook that fetches Prometheus metrics, renders alert graphs with gonum/plot, stores the images in S3‑compatible object storage, and embeds them in DingTalk notifications, providing deployment instructions, template customization, and core implementation details.

DevOps Cloud Academy
DevOps Cloud Academy
DevOps Cloud Academy
Promoter: Rendering AlertManager Graphs for DingTalk Notifications Using Go

Previously a simple Python DingTalk receiver for AlertManager was used, but it required unstable web‑scraping to capture Prometheus graphs. The new approach, implemented in Go, directly renders the alert chart, uploads it to an S3‑compatible object store, and displays the image in DingTalk messages.

The default template resides at template/default.tmpl and can be customized to fit specific notification styles, referencing the open‑source project prometheus-webhook-dingtalk .

{{ define "__subject" }}[{{ .Status | toUpper }}{{ if eq .Status "firing" }}:{{ .Alerts.Firing | len }}{{ end }}] {{ .GroupLabels.SortedPairs.Values | join " " }} {{ if gt (len .CommonLabels) (len .GroupLabels) }}({{ with .CommonLabels.Remove .GroupLabels.Names }}{{ .Values | join " " }}{{ end }}){{ end }}{{ end }}
{{ define "__alertmanagerURL" }}{{ .ExternalURL }}/#/alerts?receiver={{ .Receiver }}{{ end }}
{{ define "default.__text_alert_list" }}{{ range . }}
### {{ .Annotations.summary }}

**详情:** {{ .Annotations.description }}

{{ range .Images }}
**条件:** `{{ .Title }}`
![📈]({{ .Url }})
{{- end }}

**标签:**
{{ range .Labels.SortedPairs }}{{ if and (ne (.Name) "severity") (ne (.Name) "summary") }}> - {{ .Name }}: {{ .Value | markdown | html }}
{{ end }}{{ end }}
{{ end }}{{ end }}
{{ define "default.title" }}{{ template "__subject" . }}{{ end }}
{{ define "default.content" }}
{{ if gt (len .Alerts.Firing) 0 -}}
#### **{{ .Alerts.Firing | len }} 条报警**
{{ template "default.__text_alert_list" .Alerts.Firing }}
{{ range .AtMobiles }}@{{ . }}{{ end }}
{{- end }}
{{ if gt (len .Alerts.Resolved) 0 -}}
#### **{{ .Alerts.Resolved | len }} 条报警恢复**
{{ template "default.__text_alert_list" .Alerts.Resolved }}
{{ range .AtMobiles }}@{{ . }}{{ end }}
{{- end }}
{{- end }}

The default configuration file ( /etc/promoter/config.yaml ) defines debug mode, HTTP port, Prometheus endpoint, metric resolution, S3 credentials, and DingTalk webhook settings. The service can be run via the Docker image cnych/promoter:v0.1.1 or deployed in Kubernetes using deploy/kubernetes/promoter.yaml .

debug: true
http_port: 8080
timeout: 5s
prometheus_url:
# Prometheus address
metric_resolution: 100

s3:
  access_key:
secret_key:
endpoint: oss-cn-beijing.aliyuncs.com
  region: cn-beijing
  bucket:
dingtalk:
  url: https://oapi.dingtalk.com/robot/send?access_token=
secret:
# secret for signature

After starting Promoter, configure AlertManager to point to the webhook URL (e.g., http://promoter.kube-mon.svc.cluster.local:8080/webhook ) in the receivers section.

route:
  group_by: ['alertname','cluster']
  group_wait: 30s
  group_interval: 2m
  repeat_interval: 1h
  receiver: webhook

receivers:
- name: 'webhook'
  webhook_configs:
  - url: 'http://promoter.kube-mon.svc.cluster.local:8080/webhook'  # configure promoter webhook
    send_resolved: true

The core implementation uses the Prometheus Go client to query metric ranges and the gonum.org/v1/plot library to draw charts. The Metrics function creates a client, performs a range query, and returns a matrix of samples. The PlotMetric function builds a plot, applies a color palette, draws lines for each series, adds a polygon for threshold visualization, and renders the final PNG canvas.

func Metrics(server, query string, queryTime time.Time, duration, step time.Duration) (promModel.Matrix, error) {
    client, err := prometheus.NewClient(prometheus.Config{Address: server})
    if err != nil {
        return nil, fmt.Errorf("failed to create Prometheus client: %v", err)
    }
    api := prometheusApi.NewAPI(client)
    value, _, err := api.QueryRange(context.Background(), query, prometheusApi.Range{Start: queryTime.Add(-duration), End: queryTime, Step: duration / step})
    if err != nil {
        return nil, fmt.Errorf("failed to query Prometheus: %v", err)
    }
    metrics, ok := value.(promModel.Matrix)
    if !ok {
        return nil, fmt.Errorf("unsupported result format: %s", value.Type().String())
    }
    return metrics, nil
}

func PlotMetric(metrics promModel.Matrix, level float64, direction string) (io.WriterTo, error) {
    p, err := plot.New()
    if err != nil {
        return nil, fmt.Errorf("failed to create new plot: %v", err)
    }
    // ... (font setup, axis configuration, line drawing, polygon overlay, canvas rendering) ...
    return c, nil
}

For more details, refer to the GitHub repository cnych/promoter .

monitoringGoPrometheusS3AlertmanagerWebhookDingTalk
DevOps Cloud Academy
Written by

DevOps Cloud Academy

Exploring industry DevOps practices and technical expertise.

0 followers
Reader feedback

How this landed with the community

login Sign in to like

Rate this article

Was this worth your time?

Sign in to rate
Discussion

0 Comments

Thoughtful readers leave field notes, pushback, and hard-won operational detail here.