Implementing Observability and Alerting with Grafana Unified Alerting in a Cloud‑Native Service Mesh

1. Project Background: AutoHome is rapidly migrating its business lines (selection, news, purchase) to a cloud‑native service mesh (Istio) to improve scalability and stability, currently connecting over 200 applications and handling more than 1.5 billion mesh‑level requests per day.

2. Observability Architecture: The team built an observability stack using OpenTelemetry, Jaeger, Prometheus, and Grafana, providing automatic collection of mesh traffic, service performance (QPS, latency, P99), and end‑to‑end tracing without additional development.

3. Grafana Alert Module Introduction: Since Grafana 8.0, the unified alerting module offers visual rule creation, multi‑data‑source support, multi‑dimensional alerts, various notification channels (email, Slack, DingTalk, webhook), alert history, custom templates, and silencing capabilities.

4. Core Concepts:

Alert rules – define evaluation criteria, queries, thresholds, and evaluation intervals.

Labels – link alerts to notification policies and silences.

Notification policies – match alerts via label selectors and route them to contact points.

Contact points – specify how alerts are delivered (e.g., DingTalk, email, webhook).

5. Alert Workflow: Users create alert rules in the Grafana UI, which generate alert instances that transition through Normal, Pending, and Firing states, with history and templated messages.

6. Installation & Configuration:

# Enable unified alerting
[unified_alerting]
enabled = true

# High‑availability settings
ha_listen_address = "${POD_IP}:9094"
ha_advertise_address = "${POD_IP}:9094"
ha_peers = 10.23.2.32:9094,10.23.2.33:9094,10.23.2.34:9094

7. Practical Case – Vehicle Service Monitoring:

7.1 Configure Prometheus data source in Grafana (UI screenshot omitted).

7.2 Define an alert rule to trigger when the 99th‑percentile response time of the vehicle_service exceeds 70 ms over a 5‑minute window:

round(histogram_quantile(0.99, sum(irate(http_server_duration_bucket{service=~"vehicle_service",http_route!="/**",http_status_code="200"}[5m])) by (service,http_route,http_method,http_status_code, le)) > 60,0.01)

7.3 Add a label name=vehicle_service-rt99 to associate the rule with a notification policy.

7.4 Configure a webhook contact point:

http://alert-webhook.zhijiajishu.com/mc/multiMessage?serviceName=vehicle_service&channels=dingding

7.5 Create a custom message template using Go templating to format firing and resolved alerts.

7.6 Set a notification policy that matches the label name=vehicle_service-rt99 and routes alerts to the webhook contact point.

7.7 Define a silence schedule to suppress alerts during maintenance windows.

8. Example Alert Message (generated by the template):

[AutoMesh报警] CarAPI 99%请求的的平均处理时间超阈值报警
报警详情：应用名[vehicle_service] 接口 [/v1/app/getVehicleList],5分钟内接口平均响应时间为[79.79] 超过阈值[70ms].

9. Conclusion: The article demonstrates the end‑to‑end setup of Grafana unified alerting for a cloud‑native service mesh, covering installation, data source integration, rule definition, contact points, templating, and silencing, and points readers to official documentation for deeper exploration.