蓝绿部署与金丝雀发布:零停机部署策略实战

深入讲解蓝绿部署、金丝雀发布、滚动更新等部署策略的原理与实现,提供Kubernetes、Argo Rollouts、Flagger的实战配置,详解自动化回滚与渐进式交付最佳实践。

引言

部署是软件交付的最后一公里,也是最关键的环节。不当的部署策略可能导致服务中断、用户流失。本文将系统介绍各种零停机部署策略,并提供生产级实现方案。

部署策略对比

策略停机时间回滚速度资源消耗复杂度适用场景
重建部署开发环境
滚动更新大多数场景
蓝绿部署关键业务
金丝雀发布高风险变更
A/B测试功能验证

滚动更新(Rolling Update)

Kubernetes原生滚动更新

apiVersion: apps/v1
kind: Deployment
metadata:
  name: web-app
spec:
  replicas: 10
  strategy:
    type: RollingUpdate
    rollingUpdate:
      # 最大不可用Pod数(可以是数字或百分比)
      maxUnavailable: 1  # 或 10%
      # 最大超出Pod数
      maxSurge: 2  # 或 20%
  
  selector:
    matchLabels:
      app: web-app
  
  template:
    metadata:
      labels:
        app: web-app
    spec:
      containers:
        - name: web-app
          image: myapp:v1.0.0
          ports:
            - containerPort: 8080
          
          # 就绪探针:确保只有健康的Pod接收流量
          readinessProbe:
            httpGet:
              path: /health
              port: 8080
            initialDelaySeconds: 5
            periodSeconds: 5
            failureThreshold: 3
          
          # 存活探针:检测死锁或hang
          livenessProbe:
            httpGet:
              path: /health
              port: 8080
            initialDelaySeconds: 15
            periodSeconds: 10
          
          # 启动探针:慢启动应用
          startupProbe:
            httpGet:
              path: /health
              port: 8080
            failureThreshold: 30
            periodSeconds: 2
# 执行滚动更新
kubectl set image deployment/web-app web-app=myapp:v2.0.0

# 查看更新状态
kubectl rollout status deployment/web-app

# 回滚到上一版本
kubectl rollout undo deployment/web-app

# 查看历史版本
kubectl rollout history deployment/web-app

# 回滚到特定版本
kubectl rollout undo deployment/web-app --to-revision=3

优雅终止配置

spec:
  terminationGracePeriodSeconds: 60  # 优雅终止时间
  containers:
    - name: web-app
      lifecycle:
        preStop:
          exec:
            command:
              - /bin/sh
              - -c
              - |
                # 等待负载均衡器更新
                sleep 15
                # 执行清理操作
                /app/cleanup.sh

蓝绿部署(Blue-Green)

基本原理

初始状态:
  Blue (v1) ← 生产流量
  Green (v2) ← 待机

部署过程:
  1. 部署Green版本(v2)
  2. 测试Green版本
  3. 切换流量到Green
  4. Blue变为待机(可回滚)
  
回滚:
  立即切换回Blue

Kubernetes实现

# blue-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: web-app-blue
spec:
  replicas: 5
  selector:
    matchLabels:
      app: web-app
      version: blue
  template:
    metadata:
      labels:
        app: web-app
        version: blue
    spec:
      containers:
        - name: web-app
          image: myapp:v1.0.0
---
# green-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: web-app-green
spec:
  replicas: 5
  selector:
    matchLabels:
      app: web-app
      version: green
  template:
    metadata:
      labels:
        app: web-app
        version: green
    spec:
      containers:
        - name: web-app
          image: myapp:v2.0.0
---
# service.yaml
apiVersion: v1
kind: Service
metadata:
  name: web-app
spec:
  selector:
    app: web-app
    version: blue  # 指向当前活跃版本
  ports:
    - port: 80
      targetPort: 8080
#!/bin/bash
# deploy-blue-green.sh

# 1. 部署Green版本
kubectl apply -f green-deployment.yaml

# 2. 等待Green就绪
kubectl rollout status deployment/web-app-green

# 3. 测试Green版本
GREEN_IP=$(kubectl get pods -l version=green -o jsonpath='{.items[0].status.podIP}')
curl -f http://$GREEN_IP:8080/health
if [ $? -ne 0 ]; then
    echo "Green版本测试失败,中止部署"
    exit 1
fi

# 4. 切换流量到Green
kubectl patch service web-app -p '{"spec":{"selector":{"version":"green"}}}'

# 5. 等待一段时间观察
sleep 60

# 6. 如果一切正常,删除Blue版本
kubectl delete deployment web-app-blue

自动化蓝绿部署脚本

#!/usr/bin/env python3
import subprocess
import time
import sys

class BlueGreenDeployer:
    def __init__(self, app_name: str, namespace: str = "default"):
        self.app_name = app_name
        self.namespace = namespace
    
    def deploy(self, new_image: str, target_color: str):
        """执行蓝绿部署"""
        current_color = self.get_current_color()
        
        print(f"当前活跃版本: {current_color}")
        print(f"部署新版本到: {target_color}")
        
        # 1. 部署新版本
        self.apply_deployment(target_color, new_image)
        
        # 2. 等待就绪
        if not self.wait_for_ready(target_color):
            print("部署失败,新版本未就绪")
            return False
        
        # 3. 健康检查
        if not self.health_check(target_color):
            print("健康检查失败,中止部署")
            self.rollback_deployment(target_color)
            return False
        
        # 4. 切换流量
        self.switch_traffic(target_color)
        print(f"流量已切换到 {target_color}")
        
        # 5. 观察期
        print("进入观察期(60秒)...")
        time.sleep(60)
        
        # 6. 验证
        if not self.verify_deployment():
            print("部署后验证失败,回滚")
            self.switch_traffic(current_color)
            return False
        
        # 7. 清理旧版本
        self.cleanup_deployment(current_color)
        print("部署成功!")
        return True
    
    def get_current_color(self) -> str:
        """获取当前活跃的颜色"""
        result = subprocess.run(
            ["kubectl", "get", "service", self.app_name, 
             "-o", "jsonpath={.spec.selector.version}",
             "-n", self.namespace],
            capture_output=True, text=True
        )
        return result.stdout.strip()
    
    def apply_deployment(self, color: str, image: str):
        """应用部署配置"""
        manifest = f"""
apiVersion: apps/v1
kind: Deployment
metadata:
  name: {self.app_name}-{color}
spec:
  replicas: 5
  selector:
    matchLabels:
      app: {self.app_name}
      version: {color}
  template:
    metadata:
      labels:
        app: {self.app_name}
        version: {color}
    spec:
      containers:
        - name: {self.app_name}
          image: {image}
"""
        subprocess.run(
            ["kubectl", "apply", "-f", "-", "-n", self.namespace],
            input=manifest, text=True
        )
    
    def wait_for_ready(self, color: str, timeout: int = 300) -> bool:
        """等待部署就绪"""
        result = subprocess.run(
            ["kubectl", "rollout", "status", 
             f"deployment/{self.app_name}-{color}",
             "-n", self.namespace,
             f"--timeout={timeout}s"],
            capture_output=True
        )
        return result.returncode == 0
    
    def health_check(self, color: str) -> bool:
        """执行健康检查"""
        result = subprocess.run(
            ["kubectl", "get", "pods", 
             "-l", f"version={color}",
             "-o", "jsonpath={.items[0].status.podIP}",
             "-n", self.namespace],
            capture_output=True, text=True
        )
        pod_ip = result.stdout.strip()
        
        # 测试健康端点
        import requests
        try:
            response = requests.get(f"http://{pod_ip}:8080/health", timeout=5)
            return response.status_code == 200
        except:
            return False
    
    def switch_traffic(self, color: str):
        """切换流量"""
        subprocess.run(
            ["kubectl", "patch", "service", self.app_name,
             "-p", f'{{"spec":{{"selector":{{"version":"{color}"}}}}}}',
             "-n", self.namespace]
        )
    
    def verify_deployment(self) -> bool:
        """验证部署"""
        # 检查错误率、延迟等指标
        # 这里简化为检查Pod状态
        result = subprocess.run(
            ["kubectl", "get", "pods",
             "-l", f"app={self.app_name}",
             "-o", "jsonpath={.items[*].status.phase}",
             "-n", self.namespace],
            capture_output=True, text=True
        )
        phases = result.stdout.split()
        return all(phase == "Running" for phase in phases)
    
    def rollback_deployment(self, color: str):
        """回滚部署"""
        subprocess.run(
            ["kubectl", "delete", "deployment", 
             f"{self.app_name}-{color}",
             "-n", self.namespace]
        )
    
    def cleanup_deployment(self, color: str):
        """清理旧部署"""
        self.rollback_deployment(color)

# 使用示例
deployer = BlueGreenDeployer("web-app")
success = deployer.deploy("myapp:v2.0.0", "green")
sys.exit(0 if success else 1)

金丝雀发布(Canary)

Argo Rollouts实现

# canary-rollout.yaml
apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  name: web-app
spec:
  replicas: 10
  strategy:
    canary:
      # 金丝雀步骤
      steps:
        # 部署1个Pod(10%流量)
        - setWeight: 10
        - pause: {duration: 5m}  # 观察5分钟
        
        # 增加到20%
        - setWeight: 20
        - pause: {duration: 5m}
        
        # 增加到50%
        - setWeight: 50
        - pause: {duration: 10m}
        
        # 全量发布
        - setWeight: 100
      
      # 分析模板(自动化验证)
      analysis:
        templates:
          - templateName: success-rate
        startingStep: 1  # 从第二步开始分析
      
      # 流量管理
      canaryService: web-app-canary  # 金丝雀服务
      stableService: web-app-stable  # 稳定版服务
      
      # 流量路由(使用Istio)
      trafficRouting:
        istio:
          virtualServices:
            - name: web-app-virtualservice
              routes:
                - primary
  
  selector:
    matchLabels:
      app: web-app
  
  template:
    metadata:
      labels:
        app: web-app
    spec:
      containers:
        - name: web-app
          image: myapp:v2.0.0

分析模板(自动回滚)

# analysis-template.yaml
apiVersion: argoproj.io/v1alpha1
kind: AnalysisTemplate
metadata:
  name: success-rate
spec:
  args:
    - name: service-name
  
  metrics:
    - name: success-rate
      interval: 1m
      successCondition: result[0] >= 0.95  # 成功率>=95%
      failureLimit: 3
      provider:
        prometheus:
          address: http://prometheus:9090
          query: |
            sum(rate(
              http_requests_total{service="{{args.service-name}}", status!~"5.."}[2m]
            ))
            /
            sum(rate(
              http_requests_total{service="{{args.service-name}}"}[2m]
            ))
    
    - name: latency-p99
      interval: 1m
      successCondition: result[0] <= 0.5  # P99延迟<=500ms
      failureLimit: 3
      provider:
        prometheus:
          address: http://prometheus:9090
          query: |
            histogram_quantile(0.99,
              sum(rate(
                http_request_duration_seconds_bucket{service="{{args.service-name}}"}[2m]
              )) by (le)
            )

Flagger实现(渐进式交付)

# flagger-canary.yaml
apiVersion: flagger.app/v1beta1
kind: Canary
metadata:
  name: web-app
spec:
  targetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: web-app
  
  service:
    port: 80
    targetPort: 8080
    gateways:
      - public-gateway
    hosts:
      - app.example.com
  
  analysis:
    interval: 1m
    threshold: 5  # 连续5次失败则回滚
    maxWeight: 50  # 最大金丝雀权重
    stepWeight: 10  # 每步增加10%
    
    metrics:
      - name: request-success-rate
        thresholdRange:
          min: 99
        interval: 1m
      
      - name: request-duration
        thresholdRange:
          max: 500
        interval: 1m
    
    webhooks:
      # 负载测试
      - name: load-test
        url: http://flagger-loadtester.test/
        timeout: 5m
        metadata:
          cmd: "hey -z 1m -q 10 -c 2 http://web-app-canary.test/"
      
      # 通知
      - name: slack-notification
        type: event
        url: http://flagger-slack.notifications/
        metadata:
          channel: "#deployments"

渐进式交付(Progressive Delivery)

完整流程图

代码合并
    ↓
CI流水线(构建+测试)
    ↓
部署到Staging
    ↓
自动化测试(单元+集成+E2E)
    ↓
部署到Production(Canary 10%)
    ↓
自动化分析(指标监控)
    ↓
  ├─ 通过 → 增加流量(20% → 50% → 100%)
  └─ 失败 → 自动回滚
    ↓
全量发布完成

GitOps集成

# argocd-application.yaml
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: web-app
spec:
  project: default
  
  source:
    repoURL: https://github.com/org/web-app-manifests.git
    targetRevision: HEAD
    path: production
  
  destination:
    server: https://kubernetes.default.svc
    namespace: production
  
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    
    syncOptions:
      - CreateNamespace=true
    
    # 渐进式同步
    retry:
      limit: 5
      backoff:
        duration: 5s
        factor: 2
        maxDuration: 3m

数据库迁移与部署

安全迁移策略

-- 阶段1:添加新列(向后兼容)
ALTER TABLE users ADD COLUMN email_verified BOOLEAN DEFAULT FALSE;

-- 阶段2:部署新代码(同时写入新旧列)
-- 应用代码:INSERT时同时写入新旧列

-- 阶段3:回填历史数据
UPDATE users SET email_verified = (email IS NOT NULL)
WHERE email_verified IS NULL;

-- 阶段4:部署新代码(仅使用新列)
-- 应用代码:只读写新列

-- 阶段5:删除旧列(可选)
-- ALTER TABLE users DROP COLUMN old_email_status;

总结

部署策略选择指南:

滚动更新

  • 适用:大多数无状态服务
  • 优点:简单、资源效率高
  • 缺点:回滚慢、可能版本混合

蓝绿部署

  • 适用:关键业务、需要快速回滚
  • 优点:回滚即时、可充分测试
  • 缺点:资源消耗翻倍

金丝雀发布

  • 适用:高风险变更、需要验证
  • 优点:风险可控、数据驱动
  • 缺点:复杂度高、需要监控支持

渐进式交付

  • 适用:持续部署、成熟团队
  • 优点:完全自动化、风险最小
  • 缺点:需要完整的CI/CD和监控体系

延伸阅读

继续阅读

探索更多技术文章

浏览归档,发现更多关于系统设计、工具链和工程实践的内容。

全部文章 返回首页

「backend」更多文章

  1. 幂等性设计模式:构建可靠的分布式系统
  2. WebSocket实时通信架构:从连接到百万并发的实战指南
  3. BFF架构模式:为不同前端定制专属后端服务