DevOps

DevOps 监控告警深度实战：Prometheus、Grafana 与 Alertmanager 生产配置

深入讲解 Prometheus 四种指标类型、PromQL 进阶查询、Recording Rules、Grafana 模板化仪表板、Alertmanager 路由与静默机制，并附 Thanos 与 VictoriaMetrics 生产部署方案。

Leeting Yan 7 分钟阅读 3012 字

概述

在云原生架构中，可观测性是保障系统稳定的核心支柱。本文从 Prometheus 指标模型出发，深入 PromQL 查询、告警规则、可视化模板、路由策略，最终落地 Thanos 与 VictoriaMetrics 分布式生产方案。

一、Prometheus 四种指标类型

Prometheus 定义了四种核心指标类型，理解语义差异是编写准确 PromQL 的基础。

1.1 Counter（计数器）

Counter 代表单调递增的累积值，适用于请求总量、错误总数等。永远不要直接查询 Counter 原始值，应使用 rate() 或 increase() 计算变化速率。

# 应用层暴露 Counter
http_requests_total{method="GET", status="200"} 1024

# 抓取配置
scrape_configs:
  - job_name: 'api-gateway'
    scrape_interval: 15s
    static_configs:
      - targets: ['gateway-01:8080', 'gateway-02:8080']
    relabel_configs:
      - source_labels: [__address__]
        target_label: instance

-- 每秒 HTTP 请求速率，rate() 自动处理 Counter 重置
rate(http_requests_total[5m])
-- 过去 1 小时订单创建总数
increase(orders_created_total[1h])

1.2 Gauge（仪表盘）

Gauge 代表可增可减的瞬时值，适用于内存占用、连接数等。

-- 当前节点剩余内存
node_memory_MemAvailable_bytes

-- 预测 4 小时后磁盘是否写满
predict_linear(node_filesystem_avail_bytes[6h], 4 * 3600) < 0

1.3 Histogram（直方图）

Histogram 将采样值划分到预设 bucket，采用累积分布，适合分析延迟分布。

// Go 注册 Histogram
requestDuration := prometheus.NewHistogramVec(
  prometheus.HistogramOpts{
    Name:    "http_request_duration_seconds",
    Help:    "HTTP 请求耗时分布",
    Buckets: []float64{0.005, 0.01, 0.025, 0.05, 0.1,
                        0.25, 0.5, 1, 2.5, 5, 10},
  },
  []string{"path", "status"},
)

-- P99 请求耗时
histogram_quantile(
  0.99,
  sum(rate(http_request_duration_seconds_bucket[5m])) by (le, path)
)

-- 平均耗时 = sum / count
sum(rate(http_request_duration_seconds_sum[5m])) by (path)
/ sum(rate(http_request_duration_seconds_count[5m])) by (path)

生产建议：bucket 控制在 10-15 个以内，提前压测确定关键阈值。

1.4 Summary（摘要）

Summary 在客户端计算分位数，不可跨实例聚合分位数。适合总请求量低但精度要求高的场景，如核心支付链路。

requestLatency := prometheus.NewSummaryVec(
  prometheus.SummaryOpts{
    Name:       "http_request_latency_seconds",
    Help:       "HTTP 请求延迟摘要",
    Objectives: map[float64]float64{
      0.5:  0.05, 0.95: 0.005, 0.99: 0.001,
    },
  },
  []string{"method"},
)
指标类型适用场景计算位置可聚合性服务端开销
Counter总量、次数客户端递增高低
Gauge瞬时值、状态客户端任意中低
Histogram分布、延迟服务端分位数高中
Summary精准延迟流客户端分位数低（不可跨实例聚合）高

二、PromQL 进阶查询与告警规则

2.1 向量操作与过滤

-- 范围向量
http_requests_total[5m]

-- 偏移查询：对比 1 周前
node_cpu_seconds_total[5m] offset 1w

-- 历史同比
(avg(rate(node_cpu_seconds_total{mode!="idle"}[5m]))
 - avg(rate(node_cpu_seconds_total{mode!="idle"}[5m]) offset 1w))
/ avg(rate(node_cpu_seconds_total{mode!="idle"}[5m]) offset 1w)

2.2 二元运算符与向量匹配

-- 计算内存使用率
(node_memory_MemTotal_bytes - node_memory_MemAvailable_bytes)
/ node_memory_MemTotal_bytes * 100

-- 多对一匹配
kube_pod_container_status_restarts_total
* on (pod, namespace) group_left (resource)
kube_pod_container_resource_limits{resource="memory"}

-- 无对应 Pod 的 Endpoint
kube_endpoint_address unless kube_pod_status_ready{condition="true"}

2.3 聚合操作进阶

-- Top3 CPU 使用 Pod
topk by (namespace) (3,
  sum by (namespace, pod) (
    rate(container_cpu_usage_seconds_total{namespace=~"prod-.*"}[5m])
  )
)

-- QPS > 100 且错误率 > 1% 的接口
count by (path) (
  rate(http_requests_total{status=~"5.."}[5m]) > 0.01
  and rate(http_requests_total[5m]) > 100
)

2.4 Recording Rules 预聚合

复杂且频繁查询的表达式应进行服务端预计算，降低查询延迟。

groups:
  - name: api_latency_aggregation
    interval: 30s
    rules:
      - record: job:api_request_duration_seconds:p99_5m
        expr: |
          histogram_quantile(0.99,
            sum by (job, le) (
              rate(http_request_duration_seconds_bucket[5m])
            )
          )
        labels:
          percentile: "p99"
          team: "platform"

      - record: job:api_requests_per_second:rate5m
        expr: |
          sum by (job, status_class) (rate(http_requests_total[5m]))

-- 预聚合后查询极为简洁
job:api_request_duration_seconds:p99_5m

-- 告警规则引用 recording rule
- alert: APIHighLatency
  expr: job:api_request_duration_seconds:p99_5m > 2
  for: 5m

三、Prometheus 告警规则深度配置

3.1 高可用性与延迟类告警

groups:
  - name: service_availability
    interval: 15s
    rules:
      - alert: ServiceDown
        expr: up == 0
        for: 1m
        labels:
          severity: critical
        annotations:
          summary: "服务 {{ $labels.job }} 不可达"

      - alert: HighErrorRate
        expr: |
          (sum by (job) (rate(http_requests_total{status=~"5.."}[5m]))
           / sum by (job) (rate(http_requests_total[5m]))) > 0.05
        for: 2m
        labels:
          severity: critical
        annotations:
          summary: "{{ $labels.job }} 错误率超过 5%"

      - alert: ResourceExhaustionPredicted
        expr: |
          predict_linear(node_filesystem_avail_bytes[6h], 4 * 3600) < 0
          and node_load1 > on(instance) (
            count by(instance) (node_cpu_seconds_total{mode="idle"}) * 0.8
          )
        for: 10m
        labels:
          severity: warning

3.2 基于 SLO 的多级告警策略

groups:
  - name: slo_alerts
    rules:
      - alert: CheckoutLatencyCritical
        expr: |
          histogram_quantile(0.99,
            sum by (le) (rate(checkout_request_duration_seconds_bucket[5m]))
          ) > 3
        for: 2m
        labels:
          severity: page
        annotations:
          summary: "支付链路 P99 延迟超过 3 秒"

      - alert: CacheHitRateDeclining
        expr: |
          rate(redis_keyspace_hits_total[10m])
          / (rate(redis_keyspace_hits_total[10m])
             + rate(redis_keyspace_misses_total[10m])) < 0.8
        for: 30m
        labels:
          severity: info
        annotations:
          summary: "Redis 缓存命中率低于 80%"

四、Alertmanager 路由与静默机制

4.1 核心路由配置

global:
  resolve_timeout: 5m
  slack_api_url: '<secret>'

templates:
  - '/etc/alertmanager/templates/*.tmpl'

route:
  group_by: ['alertname', 'cluster', 'service']
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h
  receiver: 'default'
  routes:
    - match:
        severity: critical
      receiver: 'pagerduty-critical'
      group_wait: 0s
      continue: true
    - match_re:
        severity: warning|info
      receiver: 'slack-platform'
      routes:
        - match:
            team: backend
          receiver: 'slack-backend'
    - match:
        severity: page
      receiver: 'oncall-phone'
      repeat_interval: 30m

receivers:
  - name: 'default'
    email_configs:
      - to: 'ops@example.com'
        smarthost: 'smtp.example.com:587'

  - name: 'pagerduty-critical'
    pagerduty_configs:
      - service_key: '<key>'
        severity: critical

  - name: 'slack-platform'
    slack_configs:
      - channel: '#platform-alerts'
        color: '{{ if eq .CommonLabels.severity "critical" }}danger{{ else }}warning{{ end }}'

inhibit_rules:
  - source_match:
      severity: 'critical'
    target_match:
      severity: 'warning'
    equal: ['alertname', 'cluster', 'service']

  - source_match:
      alertname: 'NodeDown'
    target_match_re:
      alertname: '.*'
    equal: ['instance']

4.2 告警静默与维护窗口

#!/bin/bash
# CI/CD 流水线自动创建/删除静默

ALERTMANAGER_URL="http://alertmanager.monitoring.svc:9093"

SILENCE_ID=$(curl -s -X POST "${ALERTMANAGER_URL}/api/v2/silences" \
  -H "Content-Type: application/json" \
  -d "{
    \"matchers\": [{\"name\":\"namespace\",\"value\":\"prod-checkout\",\"isRegex\":false}],
    \"startsAt\": \"$(date -u +%Y-%m-%dT%H:%M:%SZ)\",
    \"endsAt\": \"$(date -u -d '+15 minutes' +%Y-%m-%dT%H:%M:%SZ)\",
    \"createdBy\": \"ci-cd\",
    \"comment\": \"Auto-silence during deployment\"
  }" | jq -r '.silenceID')

deploy.sh

curl -s -X DELETE "${ALERTMANAGER_URL}/api/v2/silence/${SILENCE_ID}"

五、Grafana 模板化仪表板工程化

Grafana Dashboard 应采用 Dashboard as Code 模式，通过 JSON 模型和变量模板实现版本化管理。

{
  "templating": {
    "list": [
      {
        "name": "cluster",
        "type": "query",
        "query": "label_values(node_uname_info, cluster)",
        "current": {"text": "prod", "value": "prod"},
        "label": "集群"
      },
      {
        "name": "namespace",
        "type": "query",
        "query": "label_values(kube_namespace_labels{cluster=\"$cluster\"}, namespace)",
        "multi": true,
        "includeAll": true,
        "allValue": ".*",
        "label": "命名空间"
      },
      {
        "name": "pod",
        "type": "query",
        "query": "label_values(kube_pod_info{cluster=\"$cluster\", namespace=~\"$namespace\"}, pod)",
        "multi": true,
        "includeAll": true,
        "label": "Pod"
      }
    ]
  }
}

{
  "title": "容器 CPU 使用率",
  "type": "timeseries",
  "targets": [{
    "expr": "sum by (pod) (\n  rate(container_cpu_usage_seconds_total{\n    cluster=\"$cluster\", namespace=\"$namespace\",\n    pod=~\"$pod\", container!=\"\"\n  }[$__rate_interval])\n)",
    "legendFormat": "{{ pod }}"
  }],
  "fieldConfig": {
    "defaults": {
      "unit": "percentunit",
      "min": 0, "max": 1
    }
  },
  "options": {
    "tooltip": {"mode": "multi"},
    "legend": {
      "displayMode": "table",
      "calcs": ["mean", "max", "lastNotNull"]
    }
  }
}

六、Thanos 分布式监控方案

Thanos 通过 Sidecar 将多集群数据汇聚到对象存储，实现全局视图与长期存储。

# prometheus + thanos-sidecar
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: prometheus
spec:
  replicas: 2
  template:
    spec:
      containers:
        - name: prometheus
          image: prom/prometheus:v2.53.0
          args:
            - '--config.file=/etc/prometheus/prometheus.yml'
            - '--storage.tsdb.retention.time=2d'
            - '--web.enable-lifecycle'
        - name: thanos-sidecar
          image: thanosio/thanos:v0.35.0
          args:
            - sidecar
            - '--tsdb.path=/prometheus'
            - '--prometheus.url=http://localhost:9090'
            - '--objstore.config-file=/etc/thanos/bucket.yml'
            - '--grpc.address=0.0.0.0:10901'

# bucket.yml
type: S3
config:
  bucket: "thanos-metrics"
  endpoint: "s3.amazonaws.com"
  region: "us-east-1"
  access_key: "${AWS_ACCESS_KEY}"
  secret_key: "${AWS_SECRET_KEY}"
  put_user_metadata:
    x-amz-server-side-encryption: AES256

# thanos-query - 全局查询层
apiVersion: apps/v1
kind: Deployment
metadata:
  name: thanos-query
spec:
  replicas: 3
  template:
    spec:
      containers:
        - name: query
          image: thanosio/thanos:v0.35.0
          args:
            - query
            - '--http.address=0.0.0.0:9090'
            - '--query.timeout=2m'
            - '--query.replica-label=prometheus_replica'
            - '--store=thanos-sidecar.monitoring.svc.cluster.local:10901'
            - '--store=thanos-store-gateway.monitoring.svc.cluster.local:10901'

# thanos-store-gateway - 对象存储查询网关
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: thanos-store-gateway
spec:
  replicas: 2
  template:
    spec:
      containers:
        - name: store
          image: thanosio/thanos:v0.35.0
          args:
            - store
            - '--objstore.config-file=/etc/thanos/bucket.yml'
            - '--index-cache.config-file=/etc/thanos/cache.yml'

# thanos-compactor - 压缩与降采样
apiVersion: apps/v1
kind: Deployment
metadata:
  name: thanos-compactor
spec:
  replicas: 1
  template:
    spec:
      containers:
        - name: compactor
          image: thanosio/thanos:v0.35.0
          args:
            - compact
            - '--objstore.config-file=/etc/thanos/bucket.yml'
            - '--retention.resolution-raw=30d'
            - '--retention.resolution-5m=120d'
            - '--retention.resolution-1h=1y'
            - '--compact.concurrency=4'

# thanos-ruler - 全局告警评估
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: thanos-ruler
spec:
  replicas: 2
  template:
    spec:
      containers:
        - name: ruler
          image: thanosio/thanos:v0.35.0
          args:
            - rule
            - '--rule-file=/etc/thanos/rules/*.yml'
            - '--query=thanos-query.monitoring.svc.cluster.local:10901'
            - '--alertmanagers.url=http://alertmanager.monitoring.svc.cluster.local:9093'
            - '--label=ruler_cluster="thanos-ha"'

七、VictoriaMetrics 高性能替代方案

VictoriaMetrics 兼容 Prometheus 生态，在写入吞吐和查询性能上有数量级优势。

# 单机版部署
apiVersion: apps/v1
kind: Deployment
metadata:
  name: victoria-metrics
spec:
  template:
    spec:
      containers:
        - name: victoriametrics
          image: victoriametrics/victoria-metrics:v1.102.0
          args:
            - '-storageDataPath=/vm-data'
            - '-retentionPeriod=6'
            - '-httpListenAddr=:8428'
            - '-search.maxQueryDuration=30s'
            - '-promscrape.config=/etc/vm/scrape.yml'

# vmagent - 远程写入代理
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: vmagent
spec:
  template:
    spec:
      containers:
        - name: vmagent
          image: victoriametrics/vmagent:v1.102.0
          args:
            - '-promscrape.config=/etc/vmagent/scrape.yml'
            - '-remoteWrite.url=http://vminsert.monitoring.svc.cluster.local:8480/insert/0/prometheus/'

# vmstorage - 存储节点
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: vmstorage
spec:
  replicas: 3
  template:
    spec:
      containers:
        - name: vmstorage
          image: victoriametrics/vmstorage:v1.102.0-cluster
          args:
            - '-storageDataPath=/vm-data'
            - '-vminsertAddr=:8400'
            - '-vmselectAddr=:8401'

---
# vminsert - 插入层
apiVersion: apps/v1
kind: Deployment
metadata:
  name: vminsert
spec:
  replicas: 4
  template:
    spec:
      containers:
        - name: vminsert
          image: victoriametrics/vminsert:v1.102.0-cluster
          args:
            - '-storageNode=vmstorage-0.vmstorage.svc.cluster.local:8400'
            - '-storageNode=vmstorage-1.vmstorage.svc.cluster.local:8400'
            - '-storageNode=vmstorage-2.vmstorage.svc.cluster.local:8400'

---
# vmselect - 查询层
apiVersion: apps/v1
kind: Deployment
metadata:
  name: vmselect
spec:
  replicas: 4
  template:
    spec:
      containers:
        - name: vmselect
          image: victoriametrics/vmselect:v1.102.0-cluster
          args:
            - '-storageNode=vmstorage-0.vmstorage.svc.cluster.local:8401'
            - '-storageNode=vmstorage-1.vmstorage.svc.cluster.local:8401'
            - '-storageNode=vmstorage-2.vmstorage.svc.cluster.local:8401'
            - '-replicationFactor=2'
维度VictoriaMetricsThanos
架构复杂度低中
写入性能极高依赖 Prometheus
查询性能高中（远程延迟）
对象存储依赖可选必需
多集群联邦内置Query + Store Gateway
运维成本低中
社区生态活跃，增长快成熟，CNCF 项目

八、告警质量优化与噪声抑制

groups:
  - name: actionable_alerts
    rules:
      # 预测性告警优于阈值告警
      - alert: DiskWillFillIn4Hours
        expr: predict_linear(node_filesystem_avail_bytes[6h], 4 * 3600) < 0
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "磁盘将在 4 小时内写满"
          action: "清理日志或扩容 PVC"

      # 使用 absent() 捕获指标缺失
      - alert: MetricsMissing
        expr: absent(up{job="payment-service"})
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "支付服务指标完全缺失"

{{ define "slack.platform.alert" }}
{{ $severity := .CommonLabels.severity }}
{{ $color := "good" }}
{{ if eq $severity "critical" }}{{ $color = "danger" }}
{{ else if eq $severity "warning" }}{{ $color = "warning" }}
{{ end }}
{
  "channel": "#{{ .CommonLabels.team }}-alerts",
  "attachments": [{
    "color": "{{ $color }}",
    "title": "{{ .GroupLabels.alertname }}",
    "fields": [
      {"title": "严重度", "value": "{{ $severity }}", "short": true},
      {"title": "firing", "value": "{{ .Alerts.Firing | len }}", "short": true}
    ]
  }]
}
{{ end }}

九、生产环境监控 checklist 与常见问题

9.1 部署 checklist

# Prometheus 自身监控
- job_name: 'prometheus-self'
  static_configs:
    - targets: ['localhost:9090']
  target_limit: 10000

# 外部标签标识来源
global:
  external_labels:
    cluster: prod-east
    replica: prom-01

# 远程写入双保险
remote_write:
  - url: "http://victoria-metrics:8428/api/v1/write"
    queue_config:
      capacity: 5000
      max_samples_per_send: 1000
      max_shards: 200
    write_relabel_configs:
      - source_labels: [__name__]
        regex: 'go_.*'
        action: drop

# WAL 启用压缩
# storage.tsdb.wal-compression: true

9.2 常见问题排查（FAQ）

Q1: Prometheus 查询超时 “context deadline exceeded” 如何解决？

使用 Recording Rules 预聚合高频查询；缩小时间范围或用降采样存储；用 -query.timeout 限制单次成本；检查是否存在未加标签过滤的全量查询。

Q2: Counter 进程重启归零后告警会误报吗？

rate() 和 increase() 内置重置检测，正常重启不会误报。频繁重置（如 CrashLoopBackOff）会导致短期失真。

Q3: Histogram 的 bucket 应如何设置？

压测确定 P50/P95/P99 分布；边界覆盖业务关注阈值；总量控制在 10-15 个以内。

Q4: Alertmanager 未收到告警如何排查？

# 确认规则已触发
curl -s "http://prometheus:9090/api/v1/rules?type=alert" | jq '.data.groups[].rules[] | select(.state=="firing")'

# 检查 Alertmanager 发现状态
curl -s "http://prometheus:9090/api/v1/alertmanagers" | jq '.data.activeAlertmanagers'

# 查看 Alertmanager 接收的告警
curl -s "http://alertmanager:9093/api/v2/alerts" | jq '.'

Q5: Thanos 与 VictoriaMetrics 如何共存？

短期高频数据使用 VictoriaMetrics；长期归档使用 Thanos + 对象存储。Grafana 配置多数据源，通过变量切换统一视图。

十、总结

构建生产级监控告警体系需要分层设计：

  • 采集层：Prometheus / vmagent 高可靠抓取，配合 relabel 过滤；
  • 存储层：Thanos 提供跨集群联邦与长期对象存储，VictoriaMetrics 提供高性能存储；
  • 查询层：PromQL 是基础，Recording Rules 预物化复杂查询；
  • 可视化层：Grafana 模板变量实现 Dashboard 工程化；
  • 告警层：Prometheus 规则定义触发，Alertmanager 分组、抑制、静默、路由。

监控体系需持续演进。从核心服务黄金指标（延迟、流量、错误、饱和度）开始，建立 On-call 反馈机制，定期清理无效规则，形成高信噪比的可观测平台。

基于 Prometheus v2.53、Grafana v11、Alertmanager v0.27、Thanos v0.35、VictoriaMetrics v1.102 编写。

继续阅读

探索更多技术文章

浏览归档，发现更多关于系统设计、工具链和工程实践的内容。

全部文章 返回首页

「DevOps」更多文章

  1. DevOps 文化与 CI/CD 进化：平台工程、DevEx 与组织变革
  2. DevOps 混沌工程：故障演练、稳健性验证与 Chaos Mesh 实践
  3. DevOps 日志聚合架构：ELK、Loki 与 Fluent Bit 生产部署