《后端架构设计·演进与治理》9.3 数据库变更与发布协同:expand-contract

从 arch-lab 订单服务在滚动发布中直接加非空列、旧实例写入失败的事故切入,说明 expand、migrate、contract 三阶段各自的门禁与退出条件,给出检查阶段顺序、旧代码兼容、回填完成、版本下限与安静期的 Go 实现,并附本机测试输出与边界。

一次“加个非空列”引发的下单失败

2026-11-02 02:10,arch-lab 订单服务发布 2.0.0,需要记录 courier_id。DBA 在同一窗口执行 ALTER TABLE orders ADD COLUMN courier_id BIGINT NOT NULL,发布采用滚动方式,旧版本 1.9.8 还有 12 个实例在接单。旧代码的 INSERT 不带这一列,数据库直接拒绝,下单失败持续了 6 分钟。

计划里写着“先加列,再发布”,字面上顺序没错,但加的是 NOT NULL:它对旧代码并不安全。回填没有做,contract 也没有等待旧实例退出。三个独立条件被压成了一个动作,失败几乎是必然的。数据库变更与发布的协同可以对照数据库迁移流水线 和开关与数据兼容 ,本节把重点放在“哪个阶段允许做什么、什么时候允许进入下一阶段”。

三个阶段各自回答一个问题

expand 回答“新旧代码能不能同时跑”,migrate 回答“数据有没有补齐”,contract 回答“旧代码是不是已经彻底退出”。每个阶段只做自己那类操作,跨阶段的操作即使语法上能执行,也应该被门禁拒绝。

阶段要回答的问题允许的操作禁止的操作退出条件
expand新旧代码能否同时运行加可空列、加索引、开始双写加非空约束、删列、改名、收窄类型新旧版本都已部署且能读写
migrate历史数据是否补齐回填、校验、修差异删除旧列、改旧列语义回填完成且差异为 0
contract旧代码是否已退出加非空约束、删旧列、删旧索引再改新列语义无旧版本流量,且安静期已满

改名必须拆成“加新列 + 回填 + 删旧列”,不能一步 RENAME。原因不是数据库不支持,而是滚动发布期间新旧代码会同时存在:旧代码读旧列名,新代码读新列名,任何一步改名都会让其中一侧立即失败。收窄类型同理,旧代码可能写入超出新范围的值,只有等旧写入完全停止并清理完历史数据,才能收紧。

顺序是门禁,不是建议

阶段顺序只是第一道门。真正的运行时门禁有四条:migrate 阶段必须处于双写,否则回填期间的新写入会丢失;contract 阶段必须等所有运行版本都不低于 MinAppVersion;回填必须完成;旧代码最后一次写入之后必须经过 QuietPeriod。四条缺一不可。

版本比较不能用字符串。v1.10.0 按字典序小于 v1.9.0,但按语义版本它更大;如果用字符串比较,控制器会误以为旧版本还在运行,或者更糟,误以为已经退出。安静期也不能从“发布开始”算,而要从“最后一次观察到旧代码写入”算,否则回填刚结束就删列,仍可能撞上迟到的旧请求。

order.Branch 这类状态字段的收窄尤其危险:旧代码可能写入 "cancelled_by_user",新枚举只接受 "CANCELLED"。正确做法是先把旧值映射到新值并双写,确认没有旧值再收紧。跨服务的双写与幂等可以对照双写与分支抽象 。

可运行实现

下面的 release 包与 9.1、9.2 共用。Check 先做结构检查:计划非空、阶段不倒序、expand 操作对旧代码安全、改名必须拆分;再做运行时检查:migrate 需要双写,contract 需要版本下限、回填完成和安静期。所有问题一次性返回,而不是遇到第一条就停,便于发布评审一次看到全部阻塞项。

// file: internal/platform/release/migrate.go
// tier: L1
// verify: go test ./internal/platform/release/ -count=1
package release

import (
	"errors"
	"fmt"
	"strconv"
	"strings"
	"time"
)

// Phase is one step of an expand-contract migration.
type Phase string

const (
	PhaseExpand   Phase = "expand"
	PhaseMigrate  Phase = "migrate"
	PhaseContract Phase = "contract"
)

// Op is a schema operation whose code compatibility is known up front.
type Op string

const (
	OpAddNullableColumn Op = "add_nullable_column"
	OpAddIndex          Op = "add_index"
	OpBackfill          Op = "backfill"
	OpSetNotNull        Op = "set_not_null"
	OpDropColumn        Op = "drop_column"
	OpRenameColumn      Op = "rename_column"
)

// Change is one schema operation tied to a phase.
type Change struct {
	ID     string
	Column string
	Phase  Phase
	Op     Op
}

// Plan is the ordered migration plan plus the release constraints.
type Plan struct {
	Changes       []Change
	MinAppVersion string
	QuietPeriod   time.Duration
}

// State is what the release controller can observe right now.
type State struct {
	Now            time.Time
	AppVersions    []string
	Backfilled     map[string]time.Time // backfill change ID -> completion
	StoppedWriting map[string]time.Time // column -> last old-code write
	DualWrite      bool
}

func phaseRank(p Phase) int {
	switch p {
	case PhaseExpand:
		return 0
	case PhaseMigrate:
		return 1
	case PhaseContract:
		return 2
	}
	return -1
}

func oldCodeSafe(op Op) bool {
	return op == OpAddNullableColumn || op == OpAddIndex || op == OpBackfill
}

// Check applies structural rules first, then the runtime gates: dual write,
// old readers, completed backfill and the quiet period after last old write.
func Check(p Plan, st State) []error {
	var errs []error
	if len(p.Changes) == 0 {
		return []error{errors.New("release: migration plan is empty")}
	}
	if p.MinAppVersion == "" {
		errs = append(errs, errors.New("release: plan has no minimum app version"))
	}
	if p.QuietPeriod <= 0 {
		errs = append(errs, errors.New("release: plan has no quiet period"))
	}
	if st.Now.IsZero() {
		errs = append(errs, errors.New("release: state has no clock"))
	}
	seen := map[string]bool{}
	rank := -1
	for i, ch := range p.Changes {
		where := fmt.Sprintf("change[%d] %s", i, ch.ID)
		if ch.ID == "" {
			errs = append(errs, fmt.Errorf("change[%d]: id is empty", i))
		} else if seen[ch.ID] {
			errs = append(errs, fmt.Errorf("%s: duplicate id", where))
		}
		seen[ch.ID] = true
		if ch.Column == "" {
			errs = append(errs, fmt.Errorf("%s: column is empty", where))
		}
		r := phaseRank(ch.Phase)
		if r < 0 {
			errs = append(errs, fmt.Errorf("%s: unknown phase %q", where, ch.Phase))
			continue
		}
		if r < rank {
			errs = append(errs, fmt.Errorf("%s: phase %s after a later phase", where, ch.Phase))
		}
		rank = r
		if ch.Op == OpRenameColumn {
			errs = append(errs, fmt.Errorf("%s: rename must be add+backfill+drop", where))
		}
		if ch.Phase == PhaseExpand && !oldCodeSafe(ch.Op) {
			errs = append(errs, fmt.Errorf("%s: %s is not safe for old code", where, ch.Op))
		}
	}
	for _, ch := range p.Changes {
		if ch.Phase == PhaseMigrate && !st.DualWrite {
			errs = append(errs, fmt.Errorf("%s: backfill needs dual write", ch.ID))
		}
		if ch.Phase != PhaseContract {
			continue
		}
		if !versionsAtLeast(st.AppVersions, p.MinAppVersion) {
			errs = append(errs, fmt.Errorf("%s: old code still serves: %v < %s", ch.ID, st.AppVersions, p.MinAppVersion))
		}
		if missing := missingBackfills(p, st); len(missing) > 0 {
			errs = append(errs, fmt.Errorf("%s: contract waits for backfill %v", ch.ID, missing))
		}
		stopped, ok := st.StoppedWriting[ch.Column]
		if !ok {
			errs = append(errs, fmt.Errorf("%s: no stop-writing record for %s", ch.ID, ch.Column))
		} else if quiet := st.Now.Sub(stopped); quiet < p.QuietPeriod {
			errs = append(errs, fmt.Errorf("%s: quiet %s < %s", ch.ID, quiet, p.QuietPeriod))
		}
	}
	return errs
}

func missingBackfills(p Plan, st State) []string {
	var missing []string
	for _, ch := range p.Changes {
		if ch.Phase != PhaseMigrate {
			continue
		}
		if _, ok := st.Backfilled[ch.ID]; !ok {
			missing = append(missing, ch.ID)
		}
	}
	return missing
}

func versionsAtLeast(have []string, want string) bool {
	for _, v := range have {
		if compareVersion(v, want) < 0 {
			return false
		}
	}
	return true
}

// compareVersion compares dotted versions numerically: v1.10.0 > v1.9.0.
func compareVersion(a, b string) int {
	as := strings.Split(strings.TrimPrefix(a, "v"), ".")
	bs := strings.Split(strings.TrimPrefix(b, "v"), ".")
	n := len(as)
	if len(bs) > n {
		n = len(bs)
	}
	for i := 0; i < n; i++ {
		x, y := 0, 0
		if i < len(as) {
			x, _ = strconv.Atoi(as[i])
		}
		if i < len(bs) {
			y, _ = strconv.Atoi(bs[i])
		}
		if x != y {
			if x < y {
				return -1
			}
			return 1
		}
	}
	return 0
}

两个细节值得单独说明。第一,oldCodeSafe 只放行加可空列、加索引和回填;set_not_null、drop_column、rename_column 都被归到 contract 或直接拒绝。第二,版本比较逐段转整数,缺位补 0,所以 v1.10.0 大于 v1.9.0、2 等于 v2.0.0;这比字符串比较更接近发布系统实际使用的版本语义。

测试块覆盖合法计划、阶段倒序、expand 中做非空约束、改名、旧版本仍在运行、回填缺失、安静期不足和双写关闭:

// file: internal/platform/release/migrate_test.go
// tier: L1
// verify: go test ./internal/platform/release/ -count=1
package release

import (
	"strings"
	"testing"
	"time"
)

func validPlan() Plan {
	return Plan{
		Changes: []Change{
			{ID: "E1", Column: "courier_id", Phase: PhaseExpand, Op: OpAddNullableColumn},
			{ID: "M1", Column: "courier_id", Phase: PhaseMigrate, Op: OpBackfill},
			{ID: "C1", Column: "legacy_status", Phase: PhaseContract, Op: OpDropColumn},
		},
		MinAppVersion: "v1.10.0",
		QuietPeriod:   72 * time.Hour,
	}
}

func readyState(now time.Time) State {
	return State{
		Now:            now,
		AppVersions:    []string{"v1.10.0", "v1.10.1"},
		Backfilled:     map[string]time.Time{"M1": now.Add(-80 * time.Hour)},
		StoppedWriting: map[string]time.Time{"legacy_status": now.Add(-73 * time.Hour)},
		DualWrite:      true,
	}
}

func TestValidPlanPasses(t *testing.T) {
	now := time.Date(2026, 10, 31, 8, 20, 0, 0, time.UTC)
	if errs := Check(validPlan(), readyState(now)); len(errs) != 0 {
		t.Fatalf("ready plan rejected: %v", errs)
	}
	t.Logf("changes=%d min_app=%s quiet=%s", len(validPlan().Changes), validPlan().MinAppVersion, validPlan().QuietPeriod)
	for _, tc := range []struct {
		a, b string
		want int
	}{{"v1.10.0", "v1.9.0", 1}, {"1.9.0", "v1.10.0", -1}} {
		if got := compareVersion(tc.a, tc.b); got != tc.want {
			t.Fatalf("compare(%s,%s)=%d", tc.a, tc.b, got)
		}
	}
}

func TestPlanValidationRejectsUnsafeOrderAndOps(t *testing.T) {
	cases := map[string]Plan{
		"contract_first": {Changes: []Change{
			{ID: "C1", Column: "a", Phase: PhaseContract, Op: OpDropColumn},
			{ID: "E1", Column: "b", Phase: PhaseExpand, Op: OpAddNullableColumn},
		}, MinAppVersion: "v1.0.0", QuietPeriod: time.Hour},
		"not_null_in_expand": {Changes: []Change{
			{ID: "E1", Column: "a", Phase: PhaseExpand, Op: OpSetNotNull},
		}, MinAppVersion: "v1.0.0", QuietPeriod: time.Hour},
		"rename": {Changes: []Change{
			{ID: "E1", Column: "a", Phase: PhaseExpand, Op: OpRenameColumn},
		}, MinAppVersion: "v1.0.0", QuietPeriod: time.Hour},
		"empty": {},
	}
	now := time.Date(2026, 10, 31, 8, 0, 0, 0, time.UTC)
	for name, p := range cases {
		if errs := Check(p, State{Now: now}); len(errs) == 0 {
			t.Fatalf("%s accepted", name)
		} else {
			t.Logf("%s: %v", name, errs[0])
		}
	}
}

func TestContractRuntimeGates(t *testing.T) {
	now := time.Date(2026, 10, 31, 8, 20, 0, 0, time.UTC)
	cases := []struct {
		name   string
		mutate func(*State)
		expect string
	}{
		{"old_app", func(s *State) { s.AppVersions = []string{"v1.9.0"} }, "old code still serves"},
		{"no_dual_write", func(s *State) { s.DualWrite = false }, "backfill needs dual write"},
		{"not_backfilled", func(s *State) { delete(s.Backfilled, "M1") }, "contract waits for backfill"},
		{"quiet_too_short", func(s *State) { s.StoppedWriting["legacy_status"] = now.Add(-71 * time.Hour) }, "quiet"},
		{"no_stop_record", func(s *State) { delete(s.StoppedWriting, "legacy_status") }, "no stop-writing record"},
	}
	for _, tc := range cases {
		st := readyState(now)
		tc.mutate(&st)
		errs := Check(validPlan(), st)
		found := false
		for _, err := range errs {
			found = found || strings.Contains(err.Error(), tc.expect)
		}
		if !found {
			t.Fatalf("%s: errors %v missing %q", tc.name, errs, tc.expect)
		}
		t.Logf("%s: %v", tc.name, errs[0])
	}
}

本机实测

环境为 macOS darwin/arm64、Go 1.26.0;gofmt -l . 与 go vet ./... 均无输出。运行本节相关测试:

$ go test ./internal/platform/release/ -run 'TestValidPlanPasses|TestPlanValidationRejectsUnsafeOrderAndOps|TestContractRuntimeGates' -v -count=1
=== RUN   TestValidPlanPasses
    migrate_test.go:39: changes=3 min_app=v1.10.0 quiet=72h0m0s
--- PASS: TestValidPlanPasses (0.00s)
=== RUN   TestPlanValidationRejectsUnsafeOrderAndOps
    migrate_test.go:69: empty: release: migration plan is empty
    migrate_test.go:69: contract_first: change[1] E1: phase expand after a later phase
    migrate_test.go:69: not_null_in_expand: change[0] E1: set_not_null is not safe for old code
    migrate_test.go:69: rename: change[0] E1: rename must be add+backfill+drop
--- PASS: TestPlanValidationRejectsUnsafeOrderAndOps (0.00s)
=== RUN   TestContractRuntimeGates
    migrate_test.go:98: old_app: C1: old code still serves: [v1.9.0] < v1.10.0
    migrate_test.go:98: no_dual_write: M1: backfill needs dual write
    migrate_test.go:98: not_backfilled: C1: contract waits for backfill [M1]
    migrate_test.go:98: quiet_too_short: C1: quiet 71h0m0s < 72h0m0s
    migrate_test.go:98: no_stop_record: C1: no stop-writing record for legacy_status
--- PASS: TestContractRuntimeGates (0.00s)
PASS
ok  	archlab/internal/platform/release	0.274s

读数分三组。合法计划有三个变更、三个版本约束,Check 返回空错误,说明门禁全部满足;compareVersion 同时验证了 v1.10.0 > v1.9.0,字符串比较会在这里给出相反结论。结构组里,contract 排在 expand 之前被 phase expand after a later phase 拦住,expand 中做 set_not_null 被“不安全”拦住,改名被要求拆成三步。运行时组里,旧版本仍在服务、双写关闭、回填缺失、安静期只有 71 小时、没有停止写入记录,五种情况分别给出对应原因。

边界如下:这是固定时钟、手工构造状态的单元测试,0.274 秒只代表本机。真实环境还要处理 DDL 锁、在线加索引的执行时间、回填批次对主库的压力、复制延迟导致的读旧值,以及跨服务同时写同一列的顺序问题。Check 只判断“计划与观测是否满足门禁”,不执行 DDL,也不替代数据库侧的锁与超时保护。

常见错误与边界

  • 在 expand 阶段加非空约束、删列或改名。旧代码仍在运行时,这些操作会让旧写入或旧读取立即失败;expand 只做旧代码能理解的加法。
  • 用字符串比较版本。v1.10.0 按字典序小于 v1.9.0,会把“旧版本已退出”判成“仍在线”,或者反过来提前放行 contract。
  • 回填没完成就进入 contract。回填是唯一能把历史行补齐的动作,缺了它,新代码读到 NULL 或旧值,非空约束也无法加上。
  • 安静期从发布开始算。旧实例可能在新版本上线后仍处理迟到请求,安静期必须从最后一次观察到旧代码写入算起。
  • 只改 schema 不改双写。回填期间新写入仍走旧列,补完历史数据后新数据又缺,差异永远清不干净;双写与校验要一起上线。

小结

expand-contract 的价值不在于把一次 DDL 拆成三次,而在于让每一步都有明确的兼容对象和退出条件。expand 保证新旧代码能同时跑,migrate 保证历史数据补齐,contract 保证旧代码已经退出;顺序、版本下限、回填完成和安静期是四条独立门禁,任何一条缺失都不应该放行。

本节实现只检查计划与观测,不执行迁移,也不处理锁与复制延迟;这些需要数据库侧的工具和真实流量验证。三节合起来,发布工程回答的是同一个问题:这次变更引用的是哪份确定的内容,它在什么证据下被允许继续,以及当旧代码、旧数据和新 schema 同时存在时,谁必须先退场。

阅读导航:上一节:9.2 渐进式发布:金丝雀、流量比例与自动回滚 · 下一节:10.1 租户模型:隔离级别与数据分区 。

继续阅读

探索更多技术文章

浏览归档,发现更多关于系统设计、工具链和工程实践的内容。

全部文章 返回首页

「books」更多文章

  1. 《互联网产品的 AI 架构·平台化与治理》附录 C 平台化评审清单
  2. 《互联网产品的 AI 架构·平台化与治理》附录 B 能力目录与配额字段速查
  3. 《互联网产品的 AI 架构·平台化与治理》附录 A 治理角色与职责速查