1.1 分区分服与集群拓扑
前两卷造出的 aegis 一直有个隐含前提:所有玩家都在一台机器上。卷一《网络与并发内核》把 socket、编解码、KCP、时间轮、Actor 都写完了,卷二《单服架构与玩法系统》把背包、任务、公会、交易、匹配、战斗管线都装上了——但这些统统是「一个进程里」的架构。真正的 MMO 从上线第一天起就会撞上三堵墙:一台机器的 CPU 撑不住全部 CCU、一次重启会把所有在线玩家踢下线、两个不同服的朋友想一起打副本却根本不在同一个世界里。
卷三《分布式与跨服》要解决的就是这三堵墙。而解决的第一步不是写代码,是先把「拆成什么样」想清楚。拓扑没定下来,后面每一章都会各写各的:服务发现按一套假设注册节点、RPC 按另一套假设定超时、网关又按第三套假设做粘性,最后拼不到一起。所以这一节不讲协议、不写中间件,只做两件事:把「分区、分服、全球同服」三种形态摊开对比,再把集群里五类节点的职责边界定死。定完这两件事,后面十章的每一块零件才有地方安。
三种形态:分区、分服、全球同服
行业里把集群形态讲得很乱,其实落到「玩家能不能任意相遇」这一条上,只有三种:
- 分区(zone):世界被切成若干分区,玩家在进入时就绑定到一个区,不同区的玩家永远不会在同一个世界里出现。区的边界是硬隔离——这也是它的优点:一个区出故障,只影响这个区的人。
- 分服(server):每个服是一个自包含的完整世界,玩家进入时就选定了服。跨服玩法(跨服战场、跨服拍卖)要单独做一条跨服通道,普通玩法不跨服。它比分区更彻底,运维也更简单:一个服就是一套独立的进程组。
- 全球同服(global single shard):只有一个逻辑世界,任意两个玩家都可能被放进同一场对局。它必须用动态分片 + 路由层把「谁在哪」实时算出来,代价是路由层成了全局单点,必须高可用。
三种形态的对照:
| 形态 | 玩家能否任意相遇 | 是否必须路由层 | 故障隔离 | 典型代价 |
|---|---|---|---|---|
| 分区 zone | 否,同区才能 | 否 | 一个分区宕机只影响本区 | 热门区与鬼区冷热不均 |
| 分服 server | 否,跨服玩法才相遇 | 是(仅跨服时) | 一个服宕机只影响本服 | 玩家被锁在服内,跨服要单独做 |
| 全球同服 global | 是 | 是(始终) | 无硬边界,靠背压与排空兜底 | 路由层是全局单点,必须高可用 |
这张表要读出的关键,是**「能否任意相遇」与「是否需要路由层」是绑定的**:只要玩家不能任意相遇,状态归属就能在登录时静态定下来,节点不需要随时查表;一旦要求任意相遇,归属就必须在运行时动态决定,于是目录层从「可有可无」变成「必须存在且不能挂」。大部分团队其实是从分区或分服起步的——它简单、隔离好、运维省事,只有当产品明确需要「全区全服一起打」时,才值得付全球同服的复杂度。
五类节点的职责边界
定完形态,下一步是把集群拆成几类进程。aegis 的集群有五类节点,每类节点的边界由它持有的状态决定,不由代码目录决定——这条原则在卷二 1.1 已经立过,这里把它从「单服的四个逻辑层」升级成「集群的五类节点」:
| 节点 | 持有状态 | 归属粒度 | 单写者 | 横向扩展 |
|---|---|---|---|---|
| 网关 gateway | 连接:心跳、重连票据、限速窗口 | 每连接一份 | 否(每连接各持副本) | 是 |
| 场景 scene | 地图实例:坐标、AOI、刷怪 | 每实例一份 | 是 | 是 |
| 战斗 battle | 一场对局:队伍、冷却、比分 | 每对局一份 | 是 | 是 |
| 世界 world | 玩家:背包、任务、公会、社交 | 每玩家一份 | 是 | 是 |
| 目录 directory | 路由表:id → 归属节点 | 复制控制面 | 否(Leader 写、多副本读) | 是 |
逐行看它们的边界:
- 网关持连接状态——心跳时间、重连票据、限速窗口。这些状态每个连接一份,所以网关天然可复制:三台网关各持自己那份连接副本,永远不会分叉。网关不认识「玩家」也不认识「地图」,所以它可以无脑加机器扛连接。
- 场景持地图实例状态——坐标、AOI、刷怪。以实例为键,一个实例只能有一个写者,这是不可复制的硬约束。同一个人在同一块地图上的位置,只能由一台场景进程说了算。
- 战斗持一场对局的状态——队伍、冷却、比分。它和场景一样是单写者,区别在于生命周期:场景实例常驻,战斗实例打完就销毁。战斗节点因此是「用完即走」的,调度器可以按需起停。
- 世界持玩家状态——背包、任务、公会、社交。以
PlayerID为键,同一玩家只有一个写者,但不同玩家可以分到不同的世界节点。它是唯一持「跨场景持久状态」的节点。 - 目录不持任何玩家状态,只持路由表——「玩家 P 现在在哪个世界节点、实例 I 现在在哪个场景节点」。它是唯一被刻意复制的节点:多副本读、Leader 写,因为一旦目录挂了,整个集群就找不到彼此。
这五类节点里,前四类都是「自己状态的单写者」,只有目录是复制控制面。这个不对称很关键:单写者保证了状态不分叉,而复制控制面保证了路由永远查得到。 两类节点用完全不同的高可用策略——前者靠「一个实例一个进程」,后者靠「Leader 选举 + 多副本」。第 2 章讲 Leader 选举时,会专门回来处理目录这一类的所有权问题。
可运行产物:把拓扑写成表
光画图不够。节点边界如果不落成代码,半年后一定会有人把「移动」的处理挪进世界节点,因为它「看起来也是玩家发来的」。本节的可运行产物把两条判断——五类节点各持有什么、能不能复制、三种形态能不能任意相遇、要不要路由层——编码成两张常量表,并用测试锁住它们。
下面这段是 L1,go test 可以直接跑。先看 internal/topology/topology.go:
// file: internal/topology/topology.go
// tier: L1
// verify: go test ./internal/topology/
// Package topology encodes the two questions chapter one asks about a cluster:
// which node roles exist, and which of the three deployment shapes (sharded
// zones, split servers, one global shard) a title is running.
//
// The node table is the important one. A role is defined by the state it owns,
// not by the directory it lives in: gateway owns connections, scene owns a map
// instance, battle owns one match, world owns a player, directory owns the
// routing table. Every role except directory is single-writer for the state it
// holds, which is what makes the cluster shardable at all.
package topology
// Role names one kind of process in an aegis cluster.
type Role string
const (
// RoleGateway holds connection state: handshake, heartbeat, reconnect
// ticket, rate-limit window. One copy per socket, so gateways replicate
// freely and scale horizontally.
RoleGateway Role = "gateway"
// RoleScene holds map-instance state: authoritative X/Y, AOI, spawns.
// Exactly one process mutates a given instance.
RoleScene Role = "scene"
// RoleBattle holds one match's state: teams, cooldowns, round score. It is
// created per match and thrown away afterwards.
RoleBattle Role = "battle"
// RoleWorld holds a player's persistent state: bag, quest, guild, social.
// Keyed by PlayerID, so players shard across world nodes.
RoleWorld Role = "world"
// RoleDirectory holds no player state: it owns the routing table that maps
// PlayerID / InstanceID / NodeID to the node that currently owns it.
RoleDirectory Role = "directory"
)
// Ownership describes how a role's state may be placed across processes.
type Ownership string
const (
// OwnPerConnection state is copied once per open socket; two gateways each
// holding a copy cannot disagree.
OwnPerConnection Ownership = "per-connection"
// OwnPerInstance state has exactly one writer, keyed by instance ID.
OwnPerInstance Ownership = "per-instance"
// OwnPerPlayer state has exactly one writer per player, keyed by PlayerID.
OwnPerPlayer Ownership = "per-player"
// OwnReplicated state is a control-plane copy that many nodes read and one
// leader writes; it is deliberately replicated for availability.
OwnReplicated Ownership = "replicated-control-plane"
)
// NodeRole is the section 1.1 boundary table in code.
type NodeRole struct {
Role Role
Owns string
Ownership Ownership
// SingleWriter is true when exactly one process may mutate this state.
// Every role but the directory is single-writer for what it holds.
SingleWriter bool
// Horizontal is true when the role scales out by adding identical nodes
// with no coordination beyond the routing table.
Horizontal bool
}
// nodeRoles is the five-node boundary table of section 1.1.
var nodeRoles = []NodeRole{
{RoleGateway, "connection: heartbeat, reconnect ticket, rate limit", OwnPerConnection, false, true},
{RoleScene, "map instance: position, AOI, spawns", OwnPerInstance, true, true},
{RoleBattle, "one match: teams, cooldowns, score", OwnPerInstance, true, true},
{RoleWorld, "player: bag, quest, guild, social", OwnPerPlayer, true, true},
{RoleDirectory, "routing table: id -> owning node", OwnReplicated, false, true},
}
// RoleOf returns the boundary entry for a role.
func RoleOf(r Role) (NodeRole, bool) {
for _, n := range nodeRoles {
if n.Role == r {
return n, true
}
}
return NodeRole{}, false
}
// AllRoles returns the five roles in table order.
func AllRoles() []Role {
out := make([]Role, 0, len(nodeRoles))
for _, n := range nodeRoles {
out = append(out, n.Role)
}
return out
}
// Form names one of the three deployment shapes.
type Form string
const (
// FormZone is the classic "分区": the world is cut into zones, and players
// in different zones never meet. The zone boundary is a hard partition.
FormZone Form = "zone"
// FormServer is "分服": whole independent servers, each a self-contained
// world; interaction across servers happens only through explicit
// cross-server features.
FormServer Form = "server"
// FormGlobal is "全球同服": one logical world, scaled with dynamic shards
// so any two players can meet. It pays for that reach with a routing layer.
FormGlobal Form = "global"
)
// Shape is the section 1.1 comparison of the three forms.
type Shape struct {
Form Form
// MeetAnyPair is true when any two players can be placed in one match.
MeetAnyPair bool
// NeedsRouting is true when a directory/routing layer is mandatory, because
// state ownership is decided at runtime rather than fixed at login.
NeedsRouting bool
// Isolation is what the shape buys you when one part of the world fails.
Isolation string
}
// shapes is the three-form comparison table of section 1.1.
var shapes = []Shape{
{FormZone, false, false, "a zone outage is contained to that zone"},
{FormServer, false, true, "a server outage is contained to that server"},
{FormGlobal, true, true, "no hard boundary; needs backpressure and draining to contain failure"},
}
// ShapeOf returns the comparison entry for a form.
func ShapeOf(f Form) (Shape, bool) {
for _, s := range shapes {
if s.Form == f {
return s, true
}
}
return Shape{}, false
}
再配一个测试 internal/topology/topology_test.go,断言四条不变量:五类节点齐全、只有目录不是单写者、网关与世界持有不同粒度的状态、只有全球同服才能任意相遇:
// file: internal/topology/topology_test.go
// tier: L1
// verify: go test ./internal/topology/
package topology
import "testing"
func TestFiveRolesArePresent(t *testing.T) {
want := []Role{RoleGateway, RoleScene, RoleBattle, RoleWorld, RoleDirectory}
got := AllRoles()
if len(got) != len(want) {
t.Fatalf("AllRoles() has %d roles, want %d", len(got), len(want))
}
for i := range want {
if got[i] != want[i] {
t.Errorf("AllRoles()[%d] = %s, want %s", i, got[i], want[i])
}
}
}
func TestSingleWriterRoles(t *testing.T) {
// A role is single-writer when exactly one process may mutate the state it
// holds. gateway owns per-connection state (each socket has its own copy),
// and directory is a replicated control plane; both may hold more than one
// copy. scene, battle and world are the single writers.
single := map[Role]bool{
RoleGateway: false,
RoleScene: true,
RoleBattle: true,
RoleWorld: true,
RoleDirectory: false,
}
for _, r := range AllRoles() {
n, ok := RoleOf(r)
if !ok {
t.Fatalf("RoleOf(%s): not found", r)
}
if n.SingleWriter != single[r] {
t.Errorf("RoleOf(%s).SingleWriter = %v, want %v", r, n.SingleWriter, single[r])
}
}
}
func TestGatewayAndWorldOwnDifferentKeys(t *testing.T) {
gw, _ := RoleOf(RoleGateway)
world, _ := RoleOf(RoleWorld)
if gw.Ownership != OwnPerConnection {
t.Errorf("gateway ownership = %s, want per-connection", gw.Ownership)
}
if world.Ownership != OwnPerPlayer {
t.Errorf("world ownership = %s, want per-player", world.Ownership)
}
if gw.Ownership == world.Ownership {
t.Fatal("gateway and world must not share an ownership scope")
}
}
func TestEveryRoleScalesHorizontally(t *testing.T) {
for _, r := range AllRoles() {
n, _ := RoleOf(r)
if !n.Horizontal {
t.Errorf("role %s must scale out by adding nodes", r)
}
}
}
func TestOnlyGlobalShapeMeetsAnyPair(t *testing.T) {
for _, f := range []Form{FormZone, FormServer, FormGlobal} {
s, ok := ShapeOf(f)
if !ok {
t.Fatalf("ShapeOf(%s): not found", f)
}
wantMeet := f == FormGlobal
if s.MeetAnyPair != wantMeet {
t.Errorf("ShapeOf(%s).MeetAnyPair = %v, want %v", f, s.MeetAnyPair, wantMeet)
}
}
}
func TestGlobalShapeNeedsRouting(t *testing.T) {
// The global single shard is the only shape where ownership is decided at
// runtime, so it is the only one where a routing layer is mandatory.
g, _ := ShapeOf(FormGlobal)
if !g.NeedsRouting {
t.Fatal("the global shape must require a routing layer")
}
z, _ := ShapeOf(FormZone)
if z.NeedsRouting {
t.Fatal("a hard zone boundary must not require a runtime routing layer")
}
}
本机实跑结果(go test ./internal/topology/ -v):
=== RUN TestFiveRolesArePresent
--- PASS: TestFiveRolesArePresent (0.00s)
=== RUN TestSingleWriterRoles
--- PASS: TestSingleWriterRoles (0.00s)
=== RUN TestGatewayAndWorldOwnDifferentKeys
--- PASS: TestGatewayAndWorldOwnDifferentKeys (0.00s)
=== RUN TestEveryRoleScalesHorizontally
--- PASS: TestEveryRoleScalesHorizontally (0.00s)
=== RUN TestOnlyGlobalShapeMeetsAnyPair
--- PASS: TestOnlyGlobalShapeMeetsAnyPair (0.00s)
=== RUN TestGlobalShapeNeedsRouting
--- PASS: TestGlobalShapeNeedsRouting (0.00s)
PASS
ok aegis/internal/topology 0.464s
TestGlobalShapeNeedsRouting 这条断言值得单独说一句。它把「全球同服必须配路由层」这条架构判断,变成了一条可以失败的测试:将来有人想把全球同服当成「分区去掉边界」来做,测试会立刻变红。同样,TestSingleWriterRoles 守的是「谁能复制、谁不能」——哪天有人想把场景状态也做成多副本「提高可用性」,测试会先拦住他。架构判断只有变成断言,才不会被下一个人改掉。
本节小结
这一节确立了三件事:
- 集群形态只有三种,判据是「玩家能否任意相遇」。分区与分服是硬隔离、归属可在登录时静态定死;全球同服要求任意相遇,因此必须引入运行时的路由层,并承担路由层成为全局单点的代价。
- 五类节点各有明确的职责边界:网关持连接状态(可复制)、场景持地图实例状态(单写者、常驻)、战斗持对局状态(单写者、用完即走)、世界持玩家状态(按
PlayerID分片)、目录持路由表(唯一的复制控制面)。 - 边界是可执行的:「哪个节点持有什么」「哪种形态要不要路由层」被编码成两张常量表和一组件是断言,不是注释里的约定。
拓扑定下来了,但「拆」这件事本身有成本。下一节不急着搭集群,先回答一个更前置的问题:一台机器到底能扛到多少人才值得拆,拆开又换来什么、付出什么。
阅读导航:上一节:目录 · 下一节:1.2 什么时候该拆、什么时候不该 。
继续阅读
探索更多技术文章
浏览归档,发现更多关于系统设计、工具链和工程实践的内容。