四组实验的共同口径
四组实验使用同一套设计：固定样本、配对到同一条请求、成本用整数微元、延迟用毫秒、质量分落在 0 到 1。每组只改变一个因子，并同时报告成本、延迟与质量三列。数据全部来自 ai-lab 的示例样本运行结果，不是生产实测。
四组实验分别复用前几章的模块：压缩调用第 3 章的 compressContext 与事实探针审计，缓存调用第 5 章的 evaluateCacheEconomics，小模型调用第 6 章的 evaluateDistillation，批处理调用第 7 章的 classifyTask 与 compareCost。这些模块没有被修改，实验层只负责生成样本、汇总与配对。
实验一：提示压缩
样本是 36 个会话上下文，每个包含 5 条检索片段、2 到 3 条历史与 1 条工具结果。基线保留全部片段，处理组应用去重、历史摘要、相关性阈值与全局预算。质量用 5 条事实探针衡量：每个检索片段里的单号与金额必须在保留文本中逐字存活。
| 指标 | 基线（全量） | 处理组（压缩） | 变化 |
|---|---|---|---|
| 每会话成本 | 902.67 微元 | 724.67 微元 | −19.72% |
| 每会话延迟 | 230.33 ms | 212.56 ms | −7.72% |
| 质量分 | 1.0000 | 0.8667 | −13.33pp |
| 样本数 | 36 | 36 | 配对 |
成本与延迟都显著下降，但质量分也显著下降：24 个会话各有一条低分段事实被相关性阈值丢掉。压缩策略在成本上通过，在质量护栏上失败。这正是只看 token 降幅会漏掉的结果。
实验二：结果缓存
样本是 600 条请求，覆盖 chat.reply、search.summary 与 ticket.classify 三个能力。基线每次生成，处理组命中时直接返回。命中率 57.50%，误命中 7 次；误命中代价不计入推理账单，但计入质量分。
| 指标 | 基线（无缓存） | 处理组（缓存） | 变化 |
|---|---|---|---|
| 每次请求成本 | 153.33 微元 | 72.08 微元 | −52.99% |
| 每次请求延迟 | 1033.33 ms | 445.42 ms | −56.90% |
| 质量分 | 1.0000 | 0.9883 | −1.17pp |
| 样本数 | 600 | 600 | 配对 |
现金成本从 92,000 微元降到 43,245 微元。若把 7,500 微元误命中返工计入经济成本，净收益是 41,255 微元，对应经济成本下降 44.84%。两个口径都对，但回答的问题不同：账单问现金，业务问净收益。
实验三：小模型替代
样本是 4 个能力各 10 条，共 40 条。基线是教师模型，处理组是保留率 95% 到 99% 的学生模型，单价系数 0.25、延迟系数 0.42。质量按同一条样本的教师分与学生分配对。
| 指标 | 基线（教师） | 处理组（学生） | 变化 |
|---|---|---|---|
| 每条成本 | 290.00 微元 | 72.50 微元 | −75.00% |
| 每条延迟 | 515.00 ms | 216.50 ms | −57.96% |
| 质量分 | 0.8893 | 0.8573 | −3.59% |
| 样本数 | 40 | 40 | 配对 |
总体保留率 96.41%，单条最大质量差 0.0580，关键样本失败 0 条，按第 6 章的验收策略通过。但 code.review 的保留率只有 94.15%，低于 search.rerank 的 98.15%；总体通过不代表每个能力都能替换。
实验四：批处理
样本是 10,000 条商品刷新条目，其中 200 条必须在窗口前产出。基线全部走在线；处理组 200 条在线、9,800 条进入 2 小时离线窗口。质量两边都是 1，因为批处理不改变内容，只改变处理位置。
| 指标 | 基线（全部在线） | 处理组（混合） | 变化 |
|---|---|---|---|
| 每条成本 | 204.00 微元 | 186.46 微元 | −8.60% |
| 每条延迟 | 850 ms | 3,528,017 ms | 不可直接比较 |
| 质量分 | 1.0000 | 1.0000 | 0 |
| 样本数 | 10,000 | 10,000 | 配对 |
成本从 2,040,000 微元降到 1,864,600 微元，节省 175,400 微元。延迟那一行不是「变慢 414,960%」，而是「从同步等待变成窗口内产出」：两类延迟量的语义不同。批处理的收益只对可延后消费的条目成立。
实现
experiments-four.ts 复用已有模块，只新增样本生成、指标汇总与配对样本输出。summarizeRun 同时汇总基线与处理组，避免只算处理组；allocate 把离线作业的固定开销逐微元摊到条目上，保证汇总值与总量一致。
// file: src/perf/experiments-four.ts
// tier: L2
// verify: node --test test/perf/experiments-four.test.ts
import { estimateTokens } from "../prompt/context-budget.ts";
import { compressContext, auditCompression } from "./compression.ts";
import type { CompressionPolicy, FactProbe, SourceSegment } from "./compression.ts";
import { evaluateCacheEconomics } from "./cache-econ.ts";
import type { CapabilityCacheEcon } from "./cache-econ.ts";
import { evaluateDistillation } from "./distill.ts";
import type { DistillSample, StudentProfile } from "./distill.ts";
import { breakEvenItems, classifyTask, compareCost, offlineItemMicros, onlineItemMicros } from "./batch.ts";
import type { CostShape } from "./batch.ts";
import type { PairedSample } from "./experiment.ts";
export interface ArmResult {
name: string;
units: number;
totalCostMicros: number;
meanCostMicros: number;
meanLatencyMs: number;
quality: number;
}
export interface ExperimentRun {
id: string;
baseline: ArmResult;
treatment: ArmResult;
costSamples: PairedSample[];
latencySamples: PairedSample[];
qualitySamples: PairedSample[];
notes: string[];
}
export interface FourExperimentReport {
experiments: ExperimentRun[];
baselineTotalMicros: number;
treatmentTotalMicros: number;
savingMicros: number;
}
/** 三个指标的配对样本长度必须一致；基线/处理组分别汇总，避免只算处理组。 */
function summarizeRun(
baselineName: string, treatmentName: string,
costs: readonly PairedSample[], latencies: readonly PairedSample[], qualities: readonly PairedSample[],
): { baseline: ArmResult; treatment: ArmResult } {
if (costs.length === 0 || costs.length !== latencies.length || costs.length !== qualities.length) {
throw new Error("experiment: 三个指标的样本长度不一致");
}
const average = (samples: readonly PairedSample[], side: "baseline" | "treatment"): number =>
samples.reduce((sum, item) => sum + item[side], 0) / samples.length;
const arm = (name: string, side: "baseline" | "treatment"): ArmResult => {
const total = costs.reduce((sum, item) => sum + item[side], 0);
return {
name, units: costs.length, totalCostMicros: total, meanCostMicros: total / costs.length,
meanLatencyMs: average(latencies, side), quality: average(qualities, side),
};
};
return { baseline: arm(baselineName, "baseline"), treatment: arm(treatmentName, "treatment") };
}
/** 把整数总量按条目均分，余数落在前几项；保证求和与总量逐微元一致。 */
function allocate(total: number, count: number): number[] {
const base = Math.floor(total / count);
const remainder = total - base * count;
return Array.from({ length: count }, (_, index) => base + (index < remainder ? 1 : 0));
}
const COMPRESSION_POLICY: CompressionPolicy = {
maxTokens: 300, minScore: 0.4, duplicateThreshold: 0.65, recentHistory: 1, summaryChars: 50,
};
const COMPRESSION_CAPS = ["doc.summarize", "chat.reply", "search.summary", "ticket.classify"] as const;
function contextSegments(capIndex: number, context: number): SourceSegment[] {
const cap = COMPRESSION_CAPS[capIndex];
const segments: SourceSegment[] = [];
for (let index = 0; index < 5; index += 1) {
const amount = 1200 + ((capIndex * 37 + context * 11 + index * 53) % 8800);
segments.push({
id: `${cap}-${context}-r${index}`, kind: "retrieval",
text: `检索片段${index}：单号 A${capIndex}${context}${index}，金额 ${amount} 元，客户等级 ${(context % 4) + 1}，状态已确认，需在 ${(index % 6) + 2} 小时内处理。`,
score: index === 0 ? 0.95 : index === 1 ? 0.8 : index === 2 ? 0.66 : index === 3 ? 0.52 : context % 3 !== 0 ? 0.28 : 0.55,
recency: 10 - index, mustKeep: index === 0,
});
}
for (let index = 0; index < 2 + (context % 2); index += 1) {
segments.push({
id: `${cap}-${context}-h${index}`, kind: "history",
text: index === 2
? "历史记录：客户上次选择加急处理，已确认运费差额，本次续问同一订单的发票抬头。"
: "历史记录：客户此前询问过同类问题，已按标准话术答复，未留下未结事项。",
score: 0.7, recency: index,
});
}
segments.push({
id: `${cap}-${context}-t0`, kind: "tool",
text: `工具返回：可用库存 ${(context + 1) * 7} 件，预计 ${(context % 5) + 1} 天到货，供应商编号 S${capIndex}${context}。`,
score: 0.6, recency: 20,
});
return segments;
}
function compressionRun(): ExperimentRun {
const costs: PairedSample[] = [];
const latencies: PairedSample[] = [];
const qualities: PairedSample[] = [];
let beforeTokens = 0;
let afterTokens = 0;
COMPRESSION_CAPS.forEach((cap, capIndex) => {
for (let context = 0; context < 9; context += 1) {
const segments = contextSegments(capIndex, context);
const compressed = compressContext(segments, COMPRESSION_POLICY);
const probes: FactProbe[] = segments.filter((segment) => segment.kind === "retrieval").map((segment) => {
const parts = segment.text.split("，");
return { id: segment.id, text: `${parts[0]}，${parts[1]}` };
});
const audit = auditCompression(compressed, probes);
const baselineTokens = segments.reduce((sum, segment) => sum + estimateTokens(segment.text), 0);
const treatedTokens = compressed.afterTokens;
const id = `${cap}-ctx${context}`;
costs.push({ id, baseline: baselineTokens * 3, treatment: treatedTokens * 3 });
latencies.push({ id, baseline: Math.round(140 + baselineTokens * 0.3), treatment: Math.round(140 + treatedTokens * 0.3) });
qualities.push({ id, baseline: 1, treatment: audit.probes.filter((probe) => probe.found).length / probes.length });
beforeTokens += baselineTokens;
afterTokens += treatedTokens;
}
});
return {
id: "prompt-compression",
...summarizeRun("全量上下文", "压缩上下文", costs, latencies, qualities),
costSamples: costs, latencySamples: latencies, qualitySamples: qualities,
notes: [
`压缩前 ${beforeTokens} token，压缩后 ${afterTokens} token，节省 ${beforeTokens - afterTokens} token`,
`事实探针平均存活率 ${(qualities.reduce((sum, item) => sum + item.treatment, 0) / qualities.length * 100).toFixed(2)}%`,
"成本按每 token 3 微元计算；延迟按 140ms 固定开销加 0.3ms/token 估算",
],
};
}
const CACHE_CAPS: (CapabilityCacheEcon & { hitLatencyMs: number; missLatencyMs: number; falseHits: number })[] = [
{ capability: "chat.reply", requests: 300, hits: 240, generationMicros: 120, hitServeMicros: 8, lookupMicros: 2,
entries: 50, storageMicrosPerEntry: 4, invalidationEvents: 2, invalidationMicrosPerEvent: 50,
falseHitRate: 0.0125, falseHitCostMicros: 900, hitLatencyMs: 40, missLatencyMs: 900, falseHits: 3 },
{ capability: "search.summary", requests: 200, hits: 100, generationMicros: 260, hitServeMicros: 12, lookupMicros: 6,
entries: 40, storageMicrosPerEntry: 4, invalidationEvents: 1, invalidationMicrosPerEvent: 40,
falseHitRate: 0.03, falseHitCostMicros: 1500, hitLatencyMs: 60, missLatencyMs: 1500, falseHits: 3 },
{ capability: "ticket.classify", requests: 100, hits: 5, generationMicros: 40, hitServeMicros: 5, lookupMicros: 6,
entries: 45, storageMicrosPerEntry: 4, invalidationEvents: 1, invalidationMicrosPerEvent: 20,
falseHitRate: 0.2, falseHitCostMicros: 300, hitLatencyMs: 30, missLatencyMs: 500, falseHits: 1 },
];
function cacheRun(): ExperimentRun {
const costs: PairedSample[] = [];
const latencies: PairedSample[] = [];
const qualities: PairedSample[] = [];
for (const cap of CACHE_CAPS) {
const overhead = allocate(
cap.entries * cap.storageMicrosPerEntry + cap.invalidationEvents * cap.invalidationMicrosPerEvent, cap.requests,
);
let falseSeen = 0;
for (let index = 0; index < cap.requests; index += 1) {
const remainder = (index * cap.hits) % cap.requests;
const isHit = remainder < cap.hits;
const isFalse = isHit && remainder === 0 && falseSeen < cap.falseHits;
if (isFalse) falseSeen += 1;
const id = `${cap.capability}-${index}`;
costs.push({ id, baseline: cap.generationMicros, treatment: (isHit ? cap.hitServeMicros : cap.generationMicros) + cap.lookupMicros + overhead[index] });
latencies.push({ id, baseline: cap.missLatencyMs, treatment: isHit ? cap.hitLatencyMs : cap.missLatencyMs });
qualities.push({ id, baseline: 1, treatment: isFalse ? 0 : 1 });
}
}
const economics = evaluateCacheEconomics(CACHE_CAPS);
return {
id: "result-cache",
...summarizeRun("无缓存", "启用缓存", costs, latencies, qualities),
costSamples: costs, latencySamples: latencies, qualitySamples: qualities,
notes: [
`命中率 ${(economics.hitRate * 100).toFixed(2)}%，经济净收益 ${economics.netBenefitMicros} 微元`,
`现金成本 ${economics.noCacheMicros} → ${costs.reduce((sum, item) => sum + item.treatment, 0)} 微元`,
`误命中 ${CACHE_CAPS.reduce((sum, cap) => sum + cap.falseHits, 0)} 次，误命中代价 ${economics.falseHitCostMicros} 微元不计入推理账单`,
],
};
}
const DISTILL_CAPS = ["search.rerank", "chat.reply", "doc.summarize", "code.review"] as const;
const TEACHER_COST: Record<string, number> = { "search.rerank": 180, "chat.reply": 240, "doc.summarize": 320, "code.review": 420 };
const TEACHER_LATENCY: Record<string, number> = { "search.rerank": 380, "chat.reply": 460, "doc.summarize": 540, "code.review": 680 };
const STUDENT: StudentProfile = {
id: "student-economy-v1",
retention: { "search.rerank": 0.99, "chat.reply": 0.98, "doc.summarize": 0.97, "code.review": 0.95 },
difficultyPenalty: 0.015, latencyRatio: 0.42, costRatio: 0.25, contextWindow: 8192,
};
function distillRun(): ExperimentRun {
const samples: DistillSample[] = [];
DISTILL_CAPS.forEach((capability, capIndex) => {
for (let index = 0; index < 10; index += 1) {
samples.push({
id: `${capability}-${index}`, capability, difficulty: Number((index / 9).toFixed(2)),
teacherScore: Number((0.84 + ((capIndex * 4 + index * 3) % 11) / 100).toFixed(2)),
critical: capability === "code.review" && index === 9,
});
}
});
const evaluation = evaluateDistillation(samples, STUDENT);
const costs: PairedSample[] = evaluation.samples.map((result) => ({
id: result.id, baseline: TEACHER_COST[result.capability], treatment: Math.round(TEACHER_COST[result.capability] * STUDENT.costRatio),
}));
const latencies: PairedSample[] = evaluation.samples.map((result) => ({
id: result.id, baseline: TEACHER_LATENCY[result.capability], treatment: Math.round(TEACHER_LATENCY[result.capability] * STUDENT.latencyRatio),
}));
const qualities: PairedSample[] = evaluation.samples.map((result) => ({
id: result.id, baseline: result.teacherScore, treatment: result.studentScore,
}));
return {
id: "small-model",
...summarizeRun("教师模型", "学生模型", costs, latencies, qualities),
costSamples: costs, latencySamples: latencies, qualitySamples: qualities,
notes: [
`总体质量保留率 ${(evaluation.retention * 100).toFixed(2)}%，单条最大质量差 ${evaluation.maxDrop.toFixed(4)}`,
`关键样本失败 ${evaluation.criticalFailures} 条，验收结论 ${evaluation.decision.accepted ? "通过" : "不通过"}`,
`回退能力：${evaluation.fallback.disabledCapabilities.join("、") || "无"}`,
],
};
}
const BATCH_TASK = {
id: "catalog-refresh", items: 10_000, itemToleranceMs: 6 * 3_600_000, freshnessMaxAgeMs: 12 * 3_600_000,
windowMs: 2 * 3_600_000, partialSuccessAllowed: true, consumption: "deferred" as const, urgentItems: 200,
};
const BATCH_SHAPE: CostShape = {
tokenMicrosPerItem: 180, onlineCallFixedMicros: 20, offlineJobFixedMicros: 50_000, onlineRetryRate: 0.02, offlineRetryRate: 0.005,
};
function batchRun(): ExperimentRun {
const policy = { minItems: breakEvenItems(BATCH_SHAPE), capacityPerWindow: 12_000, rerunReserveRatio: 0.1 };
const decision = classifyTask(BATCH_TASK, policy);
const comparison = compareCost(decision, BATCH_SHAPE);
const onlineItem = onlineItemMicros(BATCH_SHAPE);
const offlineItem = offlineItemMicros(BATCH_SHAPE);
const overhead = allocate(BATCH_SHAPE.offlineJobFixedMicros, decision.offlineItems);
const costs: PairedSample[] = [];
const latencies: PairedSample[] = [];
const qualities: PairedSample[] = [];
for (let index = 0; index < BATCH_TASK.items; index += 1) {
const isOnline = index < decision.onlineItems;
const offlineIndex = index - decision.onlineItems;
const id = `item-${index}`;
costs.push({ id, baseline: onlineItem, treatment: isOnline ? onlineItem : offlineItem + overhead[offlineIndex] });
latencies.push({ id, baseline: 850, treatment: isOnline ? 850 : Math.round(((offlineIndex + 0.5) * BATCH_TASK.windowMs) / decision.offlineItems) });
qualities.push({ id, baseline: 1, treatment: 1 });
}
return {
id: "offline-batch",
...summarizeRun("全部在线", "紧急在线 + 离线批处理", costs, latencies, qualities),
costSamples: costs, latencySamples: latencies, qualitySamples: qualities,
notes: [
`盈亏平衡条数 ${policy.minItems}，作业 ${BATCH_TASK.items} 条，判断 ${decision.placement}`,
`在线 ${decision.onlineItems} 条，离线 ${decision.offlineItems} 条，节省 ${comparison.savingMicros} 微元（${(comparison.savingShare * 100).toFixed(2)}%）`,
"离线延迟按条目在 2 小时窗口内均匀产出估算，不能与在线逐条延迟直接比较",
],
};
}
export function runFourExperiments(): FourExperimentReport {
const experiments = [compressionRun(), cacheRun(), distillRun(), batchRun()];
const baselineTotalMicros = experiments.reduce((sum, item) => sum + item.baseline.totalCostMicros, 0);
const treatmentTotalMicros = experiments.reduce((sum, item) => sum + item.treatment.totalCostMicros, 0);
return { experiments, baselineTotalMicros, treatmentTotalMicros, savingMicros: baselineTotalMicros - treatmentTotalMicros };
}
四个实验都返回 costSamples、latencySamples 与 qualitySamples，它们不是三个互不相干的表，而是同一批 id 上的三列观测。10.3 的配对分析、置信区间与护栏判定直接消费这三个数组，不需要重新采样。
合计与运行结果
| 实验 | 样本 | 基线总成本 | 处理总成本 | 节省 | 降幅 |
|---|---|---|---|---|---|
| 提示压缩 | 36 | 32,496 | 26,088 | 6,408 | 19.72% |
| 结果缓存 | 600 | 92,000 | 43,245 | 48,755 | 52.99% |
| 小模型替代 | 40 | 11,600 | 2,900 | 8,700 | 75.00% |
| 批处理 | 10,000 | 2,040,000 | 1,864,600 | 175,400 | 8.60% |
| 合计 | 10,676 | 2,176,096 | 1,936,833 | 239,263 | 11.00% |
合计只降 11.00%，因为批处理占总成本的 93.75%，把前三项的高降幅稀释了。这也是「整体数字」最常见的骗局：它由样本量最大的那一组决定，而不是由效果最好的那一组决定。
node --test test/perf/experiments-four.test.ts
本机一次运行的真实输出摘要如下，耗时随机器变化：
ℹ tests 6
ℹ pass 6
ℹ fail 0
ℹ duration_ms 236.353125
小结
四组实验给出四种不同的取舍：压缩降本但丢事实，缓存降本但有误命中，小模型大幅降本但能力间保留率不均，批处理降本但延迟语义改变。没有一组是「全面更优」；能被推广的只有「在指定护栏内更优」的那一组。
下一节把三列结果放进显著性、置信区间与质量回退的框架，判断哪些结论能发布、哪些只能用于探索。
阅读导航：上一节：10.1 基线、假设与实验设计 · 下一节：10.3 结果解读与回归风险 。
常见错误与边界
- 把三列结果合成一个「综合分」。权重是主观的，合成后无法判断护栏是否被突破。
- 只看总额不看单条。样本量最大的实验会主导合计降幅。
- 把离线延迟与在线延迟放在同一列比较。两者语义不同，差值没有业务含义。
- 把缓存现金成本与经济净收益混用。误命中返工不显示在推理账单上，但真实发生。
- 用总体保留率代替分能力保留率。某个能力塌陷会被其他能力平均掉。
- 假设四组实验的样本可以合并。它们的单位、分布与成本结构都不同。
- 把示例数字当成生产预期，或忘记记录模块版本。样本固定、流量不固定，阈值一变结论就不可复算。
继续阅读
探索更多技术文章
浏览归档，发现更多关于系统设计、工具链和工程实践的内容。