《MMO 服务器开发》1.1 压测口径:不要只看在线人数

「在线五千人」从来不是一句承载结论。本节先把承载指标的定义钉死——要有单位、要有变坏的方向、要能说出超标后怎么坏,再列出连接数、消息吞吐、P50/P90/P99、错误率、内存、GC 暂停这张表,最后拆穿只看在线人数、只看平均值、不看错误率、忽略慢连接这四种最常见的自欺欺人。

1.1 压测口径:不要只看在线人数

前三卷把 aegis 从网络内核、单服玩法一路造到了集群与跨服,但「造出来」不等于「扛得住」。上线前必须回答一个问题:这套服务端到底能承载多少人、多少消息? 而这个问题最容易被一句错误的话糊弄过去——「我们压过了,五千人在线没崩」。

问题出在「在线」这两个字。五千个连接挂在那里、既不登录也不发消息,对服务端来说就是五千个 fd 和五千段内核缓冲区,几乎不消耗 CPU。真正压垮服务端的是消息:每一次移动、每一次技能、每一次背包查询,都要挤过那个单线程的世界循环。所以「在线人数」既没有单位、也没有变坏的方向、更说不出超标之后会怎么坏——它压根不是承载指标。

这一节不讲怎么搭压测器(那是 1.2 的事),只做一件更前置的事:把「承载指标」这个词定义清楚,列出一张真正该看的表,再点破四种最常见的自欺欺人。口径错了,压测器再快也只是在错误的问题上跑得更快。

与站内既有文章的分工

本节与站内已有的几篇文章分工如下,它们讲方法论与架构,本节讲口径如何落成可执行的断言:

站内既有文章它讲什么本节与它的分工
/do-not-only-look-at-online-users-when-load-testing-game-servers/为什么在线人数不是承载指标本节把它落成一张带单位和失败模式的指标定义表
/game-server-bot-load-testing-platform-architecture/压测平台的架构取舍本节只定义平台要采集哪些数,平台本身在 1.2 实现
/matchmaking-service-is-more-than-filling-a-room/匹配服务不只是「凑一屋人」本节把「承载」与「业务正确」分开,匹配的正确性另章讨论

什么才算一个「承载指标」

一个数字要被称为承载指标,必须同时满足三条:有单位、有变坏的方向、超标时能说出具体的失败模式。「在线人数」三条全不满足;下面这七个数字则都满足,它们才是压测报告该有的列:

指标单位它回答什么超标后怎么坏
连接数 connections个网关真正的天花板:一连接一 fd、一 accept 槽accept 队列溢出,内核直接 RST 新连接
消息吞吐 message throughputmsg/s单写者世界每秒能真正退休多少帧请求在世界循环后排队,延迟无界上涨
P50 延迟ms普通玩家做一次动作的感受即便没坏,游戏也「黏」得让人不想玩
P99 延迟ms十分之一动作落进的尾巴,玩家投诉的那个数命中判定、技能释放在高负载下变得不可信
错误率 error rate%有多少请求压根没拿到合法应答玩家掉线、丢动作;一个漂亮的 P99 会把这事盖住
常驻内存 resident memoryMiB每连接、每实体占用乘以规模最多人在线的时刻被 OOM 杀掉
GC 暂停 gc pausems(最大)一次 stop-the-world 给所有在途请求加的停顿与请求量无关的 P99 尖刺

把这张表编码下来,比写在文档里更管用——文档会过期,代码里的表能被测试守住。下面这段是 L1,go test 可以直接跑:

// file: internal/loadtest/metrics.go
// tier: L1
// verify: go test ./internal/loadtest/
// This file is the section 1.1 "what a carrying metric is" table in code. The
// whole point of chapter one is that "online players" is not a carrying metric;
// these are the ones that are. A metric qualifies only if it has a unit, a
// direction that is worse, and a failure mode when it is exceeded.
package loadtest

// Metric is one carrying metric with a unit and the failure it predicts.
type Metric struct {
	Name string
	Unit string
	// Why is what the number tells you that "online players" does not.
	Why string
	// Bad is the failure mode once the metric is pushed past its budget.
	Bad string
}

// carryingMetrics is the section 1.1 metric definition table. Order is the
// order they should appear on a load-test report.
var carryingMetrics = []Metric{
	{
		Name: "connections",
		Unit: "count",
		Why:  "the real ceiling for a gateway: one socket, one fd, one accept-queue slot",
		Bad:  "accept backlog overflows and the kernel resets new sockets",
	},
	{
		Name: "message throughput",
		Unit: "msg/s",
		Why:  "how much work the single-writer world can actually retire per second",
		Bad:  "requests queue up behind the world loop and latency grows without bound",
	},
	{
		Name: "p50 latency",
		Unit: "ms",
		Why:  "what the median player feels on a normal action",
		Bad:  "the game feels sluggish even though nothing is technically broken",
	},
	{
		Name: "p99 latency",
		Unit: "ms",
		Why:  "the tail a tenth of actions land in; the number players complain about",
		Bad:  "hit registration and skill casts feel unreliable under peak load",
	},
	{
		Name: "error rate",
		Unit: "percent",
		Why:  "how many requests never got a valid reply at all",
		Bad:  "players see disconnects and lost actions; a green p99 hides this",
	},
	{
		Name: "resident memory",
		Unit: "MiB",
		Why:  "the per-connection and per-entity footprint multiplied by scale",
		Bad:  "the process is OOM-killed right when the most players are online",
	},
	{
		Name: "gc pause",
		Unit: "ms (max)",
		Why:  "the stall a stop-the-world collection adds to every in-flight request",
		Bad:  "a spike in p99 that has nothing to do with request volume",
	},
}

// AllMetrics returns the metric definition table in report order.
func AllMetrics() []Metric { return append([]Metric(nil), carryingMetrics...) }

// MetricByName returns the definition for a metric name.
func MetricByName(name string) (Metric, bool) {
	for _, m := range carryingMetrics {
		if m.Name == name {
			return m, true
		}
	}
	return Metric{}, false
}

这张表最该读出的一点是:「连接数」和「消息吞吐」是两个互相独立的天花板。网关的连接数受限于 fd 与 accept 队列,世界循环的吞吐受限于单写者的处理速度——它们由不同的资源决定,会先后到顶,所以压测报告必须把两列分开列,不能合成一个「承载 N 人」。

四种最常见的自欺欺人

口径定义好了,接下来是它最容易在报告里被绕开的四个地方。每一个都能让一次「通过」的压测,说出一个和线上无关的结论:

误区错误说法正确做法
只看在线人数「五千人在线,服务端没问题」五千个空闲连接什么都证明不了;要测那个数量下的每秒消息数
只报平均值「平均延迟 26ms,挺好」26ms 的平均可以藏住 1.6 秒的 P99;必须报尾巴
不看错误率「吞吐稳在 3000 msg/s,压测通过」其中若有 2% 没拿到应答,真实吞吐更低且玩家丢了动作
忽略慢连接「一条每秒只读 1KB 的连接就是一条连接」慢读者会反压它的发送队列,可能拖住共享的写者

把这四条也编成表,并配一组断言,让「在线人数不是承载指标」这句判断变成一条会变红的测试——将来有人想往报告里加回「在线人数」这一列,测试先拦住他。下面这段同样是 L1:

// file: internal/loadtest/pitfalls.go
// tier: L1
// verify: go test ./internal/loadtest/
// The section 1.1 "common mistakes" table in code. Each row is a way to report
// a passing load test that says nothing about production, paired with the fix.
package loadtest

// Pitfall is one common way a load test lies to you.
type Pitfall struct {
	Name  string
	Wrong string
	Right string
}

// pitfalls is the section 1.1 "common mistakes" table.
var pitfalls = []Pitfall{
	{
		Name:  "only watching online players",
		Wrong: "5000 people are online, so the server is fine",
		Right: "5000 idle connections prove nothing; measure messages per second at that count",
	},
	{
		Name:  "reporting only the mean",
		Wrong: "average latency is 26 ms, all good",
		Right: "a 26 ms mean can hide a 1.6 s p99; always report the tail",
	},
	{
		Name:  "ignoring the error rate",
		Wrong: "throughput held at 3000 msg/s, test passed",
		Right: "if 2% of those never got a reply, the real throughput was lower and players lost actions",
	},
	{
		Name:  "leaving slow connections in the pool",
		Wrong: "one connection that reads at 1 KiB/s is just one connection",
		Right: "a slow reader back-pressures its send queue and can stall a shared writer",
	},
}

// AllPitfalls returns the pitfall table.
func AllPitfalls() []Pitfall { return append([]Pitfall(nil), pitfalls...) }

配套的断言(L1)守住三件事:每个指标都有单位与失败模式、「在线人数」不能进表、每条误区都写清了正解:

// file: internal/loadtest/metrics_test.go
// tier: L1
// verify: go test ./internal/loadtest/
package loadtest

import "testing"

func TestEveryMetricHasUnitAndFailure(t *testing.T) {
	// A number is only a carrying metric if it has a unit and a known failure
	// mode. "online players" fails both tests, which is why it is not in the
	// table.
	for _, m := range AllMetrics() {
		if m.Name == "" || m.Unit == "" {
			t.Errorf("metric %q missing name or unit", m.Name)
		}
		if m.Why == "" || m.Bad == "" {
			t.Errorf("metric %q must state why it matters and how it fails", m.Name)
		}
	}
}

func TestOnlinePlayersIsNotAMetric(t *testing.T) {
	if _, ok := MetricByName("online players"); ok {
		t.Fatal("online players must not be a carrying metric")
	}
}

func TestPitfallsAllHaveFix(t *testing.T) {
	for _, p := range AllPitfalls() {
		if p.Name == "" || p.Wrong == "" || p.Right == "" {
			t.Errorf("pitfall %q must describe both the wrong and the right way", p.Name)
		}
	}
}

本机实跑(go test ./internal/loadtest/ -run 'Metric|Online|Pitfall' -v)三条全绿。TestOnlinePlayersIsNotAMetric 这一条值得单独说:它把一个口径判断变成了一条可执行的断言。口径之争在团队里往往吵不出结果,但一条会红的测试能——「在线人数不是承载指标」从今天起不再靠谁嗓门大。

本节小结

这一节只做了一件事,但它是后面二十九节的地基:

  1. 「在线人数」不是承载指标,因为它没有单位、没有变坏的方向、说不出超标后怎么坏。承载指标必须同时具备这三条。
  2. 真正要看的是七个数:连接数、消息吞吐、P50、P99、错误率、常驻内存、GC 暂停。其中连接数与消息吞吐是两个互相独立的天花板,必须分开列。
  3. 口径要落成断言:把指标定义表与误区表编码进代码,并让测试守住「在线人数不能进表」这条判断。

口径定死了,接下来该有工具去采集这些数。下一节从零写一个可编排的机器人压测平台:机器人状态机、并发编排、错峰连接,以及结果如何落成 JSON。

阅读导航:上一节:目录 · 下一节:1.2 机器人压测平台 。

继续阅读

探索更多技术文章

浏览归档,发现更多关于系统设计、工具链和工程实践的内容。

全部文章 返回首页

「mmo」更多文章

  1. 《MMO 服务器开发》工程化与运营 · 目录
  2. 《MMO 服务器开发》9.3 上线检查清单
  3. 《MMO 服务器开发》9.2 性能预算达标