一次超时如何变成重复预占
大促前一周,arch-lab 的下单接口 p99 从 420ms 涨到 1.9s。链路并不复杂:Order 同步调用 Inventory.Reserve,再同步调用 Payment.Authorize,最后同步调用 Notification.Send 发确认短信。支付网关当天 p99 从 180ms 涨到 760ms,SDK 在超时后自动重试了一次;而 Inventory.Reserve 当时没有幂等键,同一个 order_id 被预占了两次库存。
事故复盘会上出现了两种典型主张。第一种说“全部改异步,别再让下游拖死我们”;第二种说“异步不可靠,库存和支付必须同步”。两种主张都跳过了同一个问题:这条链路上的四个交互,对一致性、延迟、失败传播和运维能力的要求并不相同,本来就不该用同一个答案。
同步与异步不是架构风格,而是每个交互各自的组合方式。Order → Inventory 需要调用方立刻知道预占是否成功,适合同步;Order → Notification 晚 30 秒发短信没有业务影响,适合异步;Order → Payment 既需要同步拿到授权结果,又需要异步推进对账与通知,适合“同步受理 + 异步完成”的混合形态。本节把这种组合判断做成可解释的评分器,而不是又一张口号表。
四个维度不是四个口号
判断一个交互该走哪条路,至少要回答四个问题。第一是一致性:调用方继续往下走之前,是否必须看到对方的效果。第二是延迟:调用方的端到端预算,是否装得下对方的 p99 加上重试。第三是失败传播:对方失败时,调用方应该立刻失败,还是可以吸收后稍后重试。第四是运维复杂度:团队是否已经有 outbox、死信队列、重放和堆积监控。
| 维度 | 同步 | 异步 | 混合 |
|---|---|---|---|
| 一致性 | 返回即生效 | 最终生效,窗口可配 | 受理即生效,完成最终一致 |
| 延迟 | 叠加下游 p99 与重试 | 只付入队成本 | 受理快,完成不占调用方预算 |
| 失败传播 | 下游失败即调用方失败 | broker 吸收,消费者重试 | 受理失败可见,完成失败进重试 |
| 运维复杂度 | 链路短,排障直观 | 需要 outbox、DLQ、堆积告警 | 两套机制都要有 |
| 典型误用 | 把可延迟动作也串进主链路 | 没有幂等与重放就上线 | 只做同步半边,没有完成态 |
这张表只能帮助讨论,不能直接给答案。真正容易吵起来的是权重:支付团队认为一致性压倒一切,通知团队认为延迟和可用性更重要。把权重显式写出来,争议就从“异步好不好”变成“这个交互的一致性权重该是 3 还是 1”,后者可以在评审会上被回答。
站内事件、命令与消息 已经区分过三类消息的语义,本节不再重复;这里只处理“同一业务链路里如何混用”。另一个常见误区是把异步当成免费的解耦:它把运行时依赖换成了运维依赖,依赖没有消失,只是从调用栈转移到了队列和消费者。
三种组合形态
同步适合“下一步依赖上一步结果”的短链路:库存预占、支付授权、优惠券核销。它的代价是调用方可用性被下游绑架,必须配超时、重试预算和熔断,这些在从同步到异步的取舍 里已经展开。
异步适合“效果可以晚到、但必须最终到达”的动作:发短信、写搜索索引、更新推荐特征、同步数仓。它的代价是重复、乱序和堆积,必须配幂等消费、死信队列和重放工具;没有这三样,异步只是把故障从请求线程挪到了没人看的地方。
混合适合“先受理、后完成”的长流程:支付授权返回后立即给用户结果,同时发事件推进对账、开票和通知。混合不是“同步调用里顺手发个消息”那么简单,它要求受理与完成各有独立的状态和补偿路径,否则完成侧失败时没人知道该退哪一步。
可运行的决策器
下面的 Recommend 把四个维度转成 0–3 分,乘以权重后比较三种形态。它不替人做决定,但会把每个维度的打分和理由打印出来,评审时可以只推翻某一行输入,而不是推翻整个结论。
// file: internal/platform/contract/decision.go
// tier: L1
// verify: go test ./internal/platform/contract/ -count=1
package contract
import (
"fmt"
"strings"
)
// Transport is how one interaction between two services is carried.
type Transport string
const (
TransportSync Transport = "sync"
TransportAsync Transport = "async"
TransportHybrid Transport = "hybrid" // sync accept + async completion
)
// Requirement is the input to the decision. Every field must be answerable
// before code is written, not a matter of taste.
type Requirement struct {
Name string
NeedsImmediateResult bool
StalenessMS int // 0 means immediate
LatencyBudgetMS int
ProviderP99MS int
FailFast bool
OpsMaturity int // 1 (no DLQ/replay) .. 5 (routine)
}
// Weights are the relative importance of the four axes.
type Weights struct{ Consistency, Latency, Failure, Ops int }
// DefaultWeights is the weighting used by the chapter.
func DefaultWeights() Weights { return Weights{3, 2, 2, 1} }
// Recommendation keeps the per-axis scores so a reviewer can disagree with
// one input instead of rejecting the whole answer.
type Recommendation struct {
Transport Transport
Scores map[Transport]int
Reasons []string
Risks []string
}
func (r Requirement) validate() error {
switch {
case strings.TrimSpace(r.Name) == "":
return fmt.Errorf("contract: requirement name is empty")
case r.LatencyBudgetMS <= 0:
return fmt.Errorf("contract: latency budget must be positive, got %d", r.LatencyBudgetMS)
case r.ProviderP99MS <= 0:
return fmt.Errorf("contract: provider p99 must be positive, got %d", r.ProviderP99MS)
case r.StalenessMS < 0:
return fmt.Errorf("contract: staleness must not be negative, got %d", r.StalenessMS)
case r.OpsMaturity < 1 || r.OpsMaturity > 5:
return fmt.Errorf("contract: ops maturity must be 1..5, got %d", r.OpsMaturity)
}
return nil
}
// Recommend scores sync, async and hybrid on consistency, latency, failure
// propagation and operational cost. Ties prefer sync, then async, then
// hybrid: the cheaper model wins when the evidence does not separate them.
func Recommend(r Requirement, w Weights) (Recommendation, error) {
if err := r.validate(); err != nil {
return Recommendation{}, err
}
if w.Consistency < 0 || w.Latency < 0 || w.Failure < 0 || w.Ops < 0 {
return Recommendation{}, fmt.Errorf("contract: weights must not be negative")
}
if w.Consistency+w.Latency+w.Failure+w.Ops == 0 {
return Recommendation{}, fmt.Errorf("contract: all weights are zero")
}
scores := map[Transport]int{}
var reasons []string
add := func(axis string, weight, sync, async, hybrid int, why string) {
if weight == 0 {
return
}
scores[TransportSync] += sync * weight
scores[TransportAsync] += async * weight
scores[TransportHybrid] += hybrid * weight
reasons = append(reasons, fmt.Sprintf("%s(w=%d) %s [s=%d a=%d h=%d]",
axis, weight, why, sync, async, hybrid))
}
switch {
case r.NeedsImmediateResult || r.StalenessMS == 0:
add("consistency", w.Consistency, 3, 0, 3, "immediate result required")
case r.StalenessMS < 1000:
add("consistency", w.Consistency, 2, 1, 3, "staleness below 1s")
default:
add("consistency", w.Consistency, 1, 3, 2, "staleness tolerated")
}
switch {
case r.LatencyBudgetMS < r.ProviderP99MS:
add("latency", w.Latency, 0, 3, 2, "budget below provider p99")
case r.LatencyBudgetMS < 2*r.ProviderP99MS:
add("latency", w.Latency, 1, 3, 2, "budget below twice provider p99")
default:
add("latency", w.Latency, 3, 3, 3, "budget fits with headroom")
}
if r.FailFast {
add("failure", w.Failure, 2, 3, 3, "failure must stop the caller")
} else {
add("failure", w.Failure, 0, 3, 2, "failure can be absorbed")
}
switch {
case r.OpsMaturity <= 2:
add("operations", w.Ops, 3, 1, 1, "no DLQ/replay/lag monitoring")
case r.OpsMaturity == 3:
add("operations", w.Ops, 3, 2, 2, "basic alerts, manual replay")
default:
add("operations", w.Ops, 2, 3, 3, "outbox, DLQ and replay are routine")
}
best := TransportSync
for _, t := range []Transport{TransportAsync, TransportHybrid} {
if scores[t] > scores[best] {
best = t
}
}
var risks []string
switch best {
case TransportAsync:
if r.OpsMaturity <= 2 {
risks = append(risks, "async with ops maturity <=2: expect silent loss")
}
if r.NeedsImmediateResult || r.StalenessMS == 0 {
risks = append(risks, "async cannot satisfy immediate-result requirement")
}
case TransportSync:
if r.LatencyBudgetMS < r.ProviderP99MS {
risks = append(risks, "sync cannot meet budget even before retries")
}
if r.StalenessMS >= 1000 && !r.NeedsImmediateResult {
risks = append(risks, "sync couples a staleness-tolerant flow")
}
case TransportHybrid:
risks = append(risks, "hybrid needs idempotent consumer and sync fallback")
}
return Recommendation{Transport: best, Scores: scores, Reasons: reasons, Risks: risks}, nil
}
// String prints the score line used by tests and CI logs.
func (r Recommendation) String() string {
return fmt.Sprintf("transport=%s sync=%d async=%d hybrid=%d",
r.Transport, r.Scores[TransportSync], r.Scores[TransportAsync], r.Scores[TransportHybrid])
}
测试用五组真实讨论过的交互,输入来自复盘会上的估算值,不是生产监控数字。它验证的是评分逻辑和并列时的取舍顺序,不是预测线上延迟。
// file: internal/platform/contract/decision_test.go
// tier: L1
// verify: go test ./internal/platform/contract/ -count=1
package contract
import "testing"
func TestRecommendOrderFlow(t *testing.T) {
cases := []struct {
name string
req Requirement
want Transport
}{
{
name: "payment authorization",
req: Requirement{
Name: "payment-authorize", NeedsImmediateResult: true,
LatencyBudgetMS: 300, ProviderP99MS: 120,
FailFast: true, OpsMaturity: 2,
},
want: TransportSync,
},
{
name: "order confirmation email",
req: Requirement{
Name: "order-email", StalenessMS: 60000,
LatencyBudgetMS: 200, ProviderP99MS: 400,
OpsMaturity: 3,
},
want: TransportAsync,
},
{
name: "checkout stock reservation",
req: Requirement{
Name: "reserve-stock", NeedsImmediateResult: true,
LatencyBudgetMS: 500, ProviderP99MS: 80,
FailFast: true, OpsMaturity: 4,
},
want: TransportHybrid,
},
{
name: "search index update",
req: Requirement{
Name: "search-index", StalenessMS: 5000,
LatencyBudgetMS: 100, ProviderP99MS: 50,
OpsMaturity: 4,
},
want: TransportAsync,
},
{
name: "coupon validation",
req: Requirement{
Name: "coupon-check", NeedsImmediateResult: true,
LatencyBudgetMS: 150, ProviderP99MS: 100,
OpsMaturity: 3,
},
want: TransportHybrid,
},
}
for _, tc := range cases {
rec, err := Recommend(tc.req, DefaultWeights())
if err != nil {
t.Fatalf("%s: unexpected error: %v", tc.name, err)
}
if rec.Transport != tc.want {
t.Errorf("%s: got %s, want %s", tc.name, rec.Transport, tc.want)
}
t.Logf("%s: %s reasons=%d risks=%d", tc.name, rec, len(rec.Reasons), len(rec.Risks))
for _, risk := range rec.Risks {
t.Logf(" risk: %s", risk)
}
}
}
func TestRecommendRejectsInvalidInput(t *testing.T) {
base := Requirement{Name: "valid", LatencyBudgetMS: 100, ProviderP99MS: 50, OpsMaturity: 3}
cases := []struct {
name string
req Requirement
w Weights
}{
{"empty name", Requirement{LatencyBudgetMS: 100, ProviderP99MS: 50, OpsMaturity: 3}, DefaultWeights()},
{"zero budget", Requirement{Name: "x", ProviderP99MS: 50, OpsMaturity: 3}, DefaultWeights()},
{"negative staleness", Requirement{Name: "x", StalenessMS: -1, LatencyBudgetMS: 100, ProviderP99MS: 50, OpsMaturity: 3}, DefaultWeights()},
{"ops out of range", Requirement{Name: "x", LatencyBudgetMS: 100, ProviderP99MS: 50, OpsMaturity: 6}, DefaultWeights()},
{"all weights zero", base, Weights{}},
}
for _, tc := range cases {
if _, err := Recommend(tc.req, tc.w); err == nil {
t.Errorf("%s: expected error, got nil", tc.name)
} else {
t.Logf("%s: rejected: %v", tc.name, err)
}
}
}
本机实测:五组交互与五种非法输入
在 .local/checks/be-arch-gov-ch02 下建立 module archlab(go 1.26.0),执行 gofmt -l .、go vet ./...、go test ./... -count=1 -v,三者均无报错。环境为 macOS darwin/arm64、Go 1.26.0。完整原始输出保存在 .local/reports/be-arch-gov/ch02-be-arch-gov-02-1-sync-async-composition.txt。
=== RUN TestRecommendOrderFlow
decision_test.go:68: payment authorization: transport=sync sync=22 async=13 hybrid=22 reasons=4 risks=0
decision_test.go:68: order confirmation email: transport=async sync=6 async=23 hybrid=16 reasons=4 risks=0
decision_test.go:68: checkout stock reservation: transport=hybrid sync=21 async=15 hybrid=24 reasons=4 risks=1
decision_test.go:70: risk: hybrid needs idempotent consumer and sync fallback
decision_test.go:68: search index update: transport=async sync=11 async=24 hybrid=19 reasons=4 risks=0
decision_test.go:68: coupon validation: transport=hybrid sync=14 async=14 hybrid=19 reasons=4 risks=1
decision_test.go:70: risk: hybrid needs idempotent consumer and sync fallback
--- PASS: TestRecommendOrderFlow (0.00s)
=== RUN TestRecommendRejectsInvalidInput
decision_test.go:92: empty name: rejected: contract: requirement name is empty
decision_test.go:92: zero budget: rejected: contract: latency budget must be positive, got 0
decision_test.go:92: negative staleness: rejected: contract: staleness must not be negative, got -1
decision_test.go:92: ops out of range: rejected: contract: ops maturity must be 1..5, got 6
decision_test.go:92: all weights zero: rejected: contract: all weights are zero
--- PASS: TestRecommendRejectsInvalidInput (0.00s)
PASS
ok archlab/internal/platform/contract 0.413s
读数分三层。第一,支付授权得到 sync=22、hybrid=22,按“同分优先更简单形态”的规则落到同步;这正是本次事故里不能改成纯异步的交互。第二,确认短信是 6/23/16,异步明显占优,因为它容忍 60 秒延迟,且失败可以重放。第三,库存预占是 21/15/24,混合胜出,因为团队已有 outbox 和重放能力,但同步受理仍然必要。
这些分数不是性能数字,只是权重与输入的函数。改变 DefaultWeights 会让结果变化,所以权重必须在架构评审里被明确批准,而不是藏在某个人的默认值里。生产上要填的 ProviderP99MS 必须来自监控,不能用本节示例值;示例值只用于演示判定方向。
常见错误与边界
第一,全站统一选型。“我们全面异步”与“我们全部同步”一样,都是把每个交互不同的约束压成一个口号。正确做法是逐交互填 Requirement,再把结果放在一张表里评审。
第二,把 broker 当可靠性。异步只保证“调用方不等”,不保证“消息一定被处理”。没有 outbox、死信队列、重放和堆积告警时,异步会把失败藏起来,事故时连丢了多少条都说不清。
第三,混合只做一半。同步受理成功、异步完成失败时,必须有补偿或人工队列;否则用户看到“已支付”,对账侧永远缺一笔。混合形态的运维成本接近异步加同步之和,不能按同步的预算排期。
第四,用平均延迟代替 p99。决策器接受的是预算和 p99,不是均值;均值 100ms、p99 900ms 的下游,在 300ms 预算下仍然是不可同步的。
小结
同步与异步不是二选一,而是每个交互各自的一次组合决策。把一致性、延迟、失败传播和运维复杂度显式打分,能让“要不要异步”变成“这四个输入填得对不对”,评审也才有可推翻的对象。
本机的五组用例给出的是判定方向:支付授权落在同步,确认短信和搜索索引落在异步,库存预占与优惠券校验落在混合;这些结果只对给定的示例输入成立。真正上线前,把 ProviderP99MS 换成监控值、把 OpsMaturity 换成经过演练验证的等级,再重跑同一组测试。
混合形态尤其需要克制:它要求同步受理和异步完成各有独立状态、幂等键与补偿路径。做不到这三点时,宁可先选同步并把超时预算做小,也不要用一个没有完成态的“半异步”方案。
阅读导航:上一节:1.3 订单域的一次拆分演练:边界、共享数据与回退 · 下一节:2.2 契约版本演进:向后兼容、弃用与消费者驱动 。
继续阅读
探索更多技术文章
浏览归档,发现更多关于系统设计、工具链和工程实践的内容。