序列数据（文本、时间序列、语音）的建模是深度学习的核心任务之一。从 RNN 的循环结构到 Transformer 的自注意力机制，架构的演进带来了并行化和长距离依赖的突破。本文完整梳理这一技术脉络。
1. RNN 基础
1.1 循环结构
RNN 维护隐藏状态，随序列逐步传递信息：
$$h_t = \tanh(W_{hh}h_{t-1} + W_{xh}x_t + b)$$
import torch
import torch.nn as nn
class SimpleRNN(nn.Module):
def __init__(self, input_size, hidden_size):
super().__init__()
self.hidden_size = hidden_size
self.i2h = nn.Linear(input_size + hidden_size, hidden_size)
self.i2o = nn.Linear(input_size + hidden_size, 10)
self.softmax = nn.LogSoftmax(dim=1)
def forward(self, input, hidden):
combined = torch.cat((input, hidden), dim=1)
hidden = torch.tanh(self.i2h(combined))
output = self.i2o(combined)
output = self.softmax(output)
return output, hidden
def initHidden(self):
return torch.zeros(1, self.hidden_size)
1.2 RNN 的局限
- 梯度消失/爆炸：长序列反向传播时梯度指数级衰减或增长
- 长距离依赖：前面状态的信息难以传递到远处时间步
- 串行计算：无法并行化，训练慢
2. LSTM：长短期记忆网络
2.1 门控机制
LSTM 通过三个门控制信息流动：遗忘门、输入门、输出门。
关键状态：
- 细胞状态 (Cell State)：$C_t$，贯穿整条序列的传送带
- 隐藏状态 (Hidden State)：$h_t$，输出到下一层
门控公式：
$$f_t = \sigma(W_f \cdot [h_{t-1}, x_t] + b_f)$$ 遗忘门
$$i_t = \sigma(W_i \cdot [h_{t-1}, x_t] + b_i)$$ 输入门
$$\tilde{C}t = \tanh(W_C \cdot [h{t-1}, x_t] + b_C)$$ 候选状态
$$C_t = f_t \odot C_{t-1} + i_t \odot \tilde{C}_t$$ 更新细胞状态
$$o_t = \sigma(W_o \cdot [h_{t-1}, x_t] + b_o)$$ 输出门
$$h_t = o_t \odot \tanh(C_t)$$ 输出隐藏状态
class LSTMCell(nn.Module):
def __init__(self, input_size, hidden_size):
super().__init__()
self.hidden_size = hidden_size
# 门控：4 组线性变换（input, forget, output, candidate）
self.gates = nn.Linear(input_size + hidden_size, 4 * hidden_size)
def forward(self, x, state):
h, c = state # hidden state, cell state
combined = torch.cat([x, h], dim=-1)
gates = self.gates(combined)
# 分割四个门
i, f, o, g = gates.chunk(4, dim=-1)
i = torch.sigmoid(i) # 输入门
f = torch.sigmoid(f) # 遗忘门
o = torch.sigmoid(o) # 输出门
g = torch.tanh(g) # 候选
c = f * c + i * g # 细胞状态更新
h = o * torch.tanh(c) # 隐藏状态输出
return h, (h, c)
2.2 PyTorch LSTM 使用
lstm = nn.LSTM(
input_size=128, # 输入特征维度
hidden_size=256, # 隐藏层维度
num_layers=2, # 堆叠层数
batch_first=True, # 输入格式 (batch, seq, feature)
dropout=0.3, # 层间 dropout
bidirectional=True # 双向 LSTM
)
# 输入: (batch=32, seq=50, features=128)
x = torch.randn(32, 50, 128)
# 输出: output (32, 50, 512), hidden (4, 32, 256), cell (4, 32, 256)
output, (hidden, cell) = lstm(x)
# output[:, -1, :] 取最后时刻的隐藏状态
# hidden 形状: (num_layers * num_directions, batch, hidden_size)
2.3 GRU：简化版 LSTM
GRU 合并细胞状态与隐藏状态，减少一个门。
$$z_t = \sigma(W_z \cdot [h_{t-1}, x_t])$$ 更新门
$$r_t = \sigma(W_r \cdot [h_{t-1}, x_t])$$ 重置门
$$\tilde{h}t = \tanh(W \cdot [r_t \odot h{t-1}, x_t])$$ 候选隐藏
$$h_t = (1 - z_t) \odot h_{t-1} + z_t \odot \tilde{h}_t$$
gru = nn.GRU(input_size=128, hidden_size=256, num_layers=2,
batch_first=True, bidirectional=True)
output, hidden = gru(x)
LSTM vs GRU：
- LSTM 更强大，参数多，小数据表现更好
- GRU 更快，参数少，大数据下两者相当
3. Seq2Seq 与注意力机制
3.1 编码器-解码器架构
输入序列 → [Encoder] → 上下文向量 → [Decoder] → 输出序列
瓶颈：固定长度的上下文向量难以编码长序列信息。
3.2 Bahdanau 注意力
为每个解码时间步动态计算输入序列的加权表示：
$$e_{ij} = v^T \tanh(W_s s_{i-1} + W_h h_j)$$ 对齐分数
$$\alpha_{ij} = \frac{\exp(e_{ij})}{\sum_k \exp(e_{ik})}$$ 注意力权重
$$c_i = \sum_j \alpha_{ij} h_j$$ 上下文向量
class Attention(nn.Module):
def __init__(self, hidden_size):
super().__init__()
self.attn = nn.Linear(hidden_size * 2, hidden_size)
self.v = nn.Parameter(torch.rand(hidden_size))
def forward(self, hidden, encoder_outputs):
# hidden: (batch, hidden)
# encoder_outputs: (batch, seq_len, hidden)
seq_len = encoder_outputs.size(1)
hidden = hidden.unsqueeze(1).repeat(1, seq_len, 1)
energy = torch.tanh(self.attn(torch.cat([hidden, encoder_outputs], dim=2)))
energy = energy.permute(0, 2, 1)
v = self.v.repeat(encoder_outputs.size(0), 1).unsqueeze(1)
attention = torch.bmm(v, energy).squeeze(1) # (batch, seq_len)
return F.softmax(attention, dim=1)
4. Transformer：Attention Is All You Need
Transformer 用 Self-Attention 完全替代循环结构，实现全局依赖的并行计算。
4.1 整体架构
输入 → [Embedding + 位置编码] → [Encoder x N] → [Decoder x N] → [Linear + Softmax] → 输出
Encoder: [Multi-Head Attention → Add&Norm → FeedForward → Add&Norm] x N
Decoder: [Masked MHA → Add&Norm → Cross MHA → Add&Norm → FF → Add&Norm] x N
4.2 Self-Attention
查询 (Query)、键 (Key)、值 (Value) 机制：
$$Attention(Q, K, V) = softmax(\frac{QK^T}{\sqrt{d_k}})V$$
import math
class SelfAttention(nn.Module):
def __init__(self, embed_dim):
super().__init__()
self.query = nn.Linear(embed_dim, embed_dim)
self.key = nn.Linear(embed_dim, embed_dim)
self.value = nn.Linear(embed_dim, embed_dim)
self.scale = math.sqrt(embed_dim)
def forward(self, x, mask=None):
# x: (batch, seq_len, embed_dim)
Q = self.query(x) # (batch, seq_len, embed_dim)
K = self.key(x)
V = self.value(x)
# 注意力分数
scores = torch.matmul(Q, K.transpose(-2, -1)) / self.scale
# scores: (batch, seq_len, seq_len)
if mask is not None:
scores = scores.masked_fill(mask == 0, float('-inf'))
attn = F.softmax(scores, dim=-1)
output = torch.matmul(attn, V)
return output, attn
4.3 多头注意力
将 Q/K/V 投影到多个子空间分别计算注意力。
class MultiHeadAttention(nn.Module):
def __init__(self, embed_dim, num_heads):
super().__init__()
assert embed_dim % num_heads == 0
self.embed_dim = embed_dim
self.num_heads = num_heads
self.head_dim = embed_dim // num_heads
self.qkv = nn.Linear(embed_dim, embed_dim * 3)
self.out_proj = nn.Linear(embed_dim, embed_dim)
def forward(self, x, mask=None):
batch_size, seq_len, _ = x.shape
# (batch, seq, 3*embed) → (batch, seq, 3, heads, head_dim)
qkv = self.qkv(x).reshape(batch_size, seq_len, 3,
self.num_heads, self.head_dim)
qkv = qkv.permute(2, 0, 3, 1, 4) # (3, batch, heads, seq, head_dim)
q, k, v = qkv[0], qkv[1], qkv[2]
# 注意力: (batch, heads, seq, seq)
scores = torch.matmul(q, k.transpose(-2, -1)) / math.sqrt(self.head_dim)
if mask is not None:
scores = scores.masked_fill(mask == 0, float('-inf'))
attn = F.softmax(scores, dim=-1)
# (batch, heads, seq, head_dim) → (batch, seq, embed)
output = torch.matmul(attn, v)
output = output.permute(0, 2, 1, 3).reshape(batch_size, seq_len, -1)
return self.out_proj(output)
4.4 位置编码
Transformer 没有循环或卷积，需显式编码位置信息。
class PositionalEncoding(nn.Module):
def __init__(self, embed_dim, max_len=5000, dropout=0.1):
super().__init__()
self.dropout = nn.Dropout(p=dropout)
position = torch.arange(max_len).unsqueeze(1)
div_term = torch.exp(torch.arange(0, embed_dim, 2) *
(-math.log(10000.0) / embed_dim))
pe = torch.zeros(max_len, embed_dim)
pe[:, 0::2] = torch.sin(position * div_term)
pe[:, 1::2] = torch.cos(position * div_term)
self.register_buffer('pe', pe.unsqueeze(0)) # (1, max_len, embed_dim)
def forward(self, x):
x = x + self.pe[:, :x.size(1)]
return self.dropout(x)
RoPE (旋转位置编码)：
现代大模型（LLaMA、GPT-NeoX）使用旋转位置编码，将位置信息融入 Q/K 的乘法中，支持外推（extrapolation）。
4.5 完整 Transformer Encoder
class TransformerEncoderLayer(nn.Module):
def __init__(self, embed_dim, num_heads, ff_dim, dropout=0.1):
super().__init__()
self.self_attn = MultiHeadAttention(embed_dim, num_heads)
self.norm1 = nn.LayerNorm(embed_dim)
self.norm2 = nn.LayerNorm(embed_dim)
self.ff = nn.Sequential(
nn.Linear(embed_dim, ff_dim),
nn.GELU(),
nn.Dropout(dropout),
nn.Linear(ff_dim, embed_dim),
nn.Dropout(dropout)
)
def forward(self, x, mask=None):
# Self-Attention + 残差 + LayerNorm
attn_out = self.self_attn(x, mask)
x = self.norm1(x + attn_out)
# FeedForward + 残差 + LayerNorm
ff_out = self.ff(x)
x = self.norm2(x + ff_out)
return x
class TransformerEncoder(nn.Module):
def __init__(self, vocab_size, embed_dim, num_heads, ff_dim,
num_layers, max_len=512, dropout=0.1):
super().__init__()
self.embedding = nn.Embedding(vocab_size, embed_dim)
self.pos_encoding = PositionalEncoding(embed_dim, max_len, dropout)
self.layers = nn.ModuleList([
TransformerEncoderLayer(embed_dim, num_heads, ff_dim, dropout)
for _ in range(num_layers)
])
self.norm = nn.LayerNorm(embed_dim)
def forward(self, x, mask=None):
x = self.embedding(x)
x = self.pos_encoding(x)
for layer in self.layers:
x = layer(x, mask)
return self.norm(x)
4.6 注意力可视化
def visualize_attention(attn_weights, tokens):
"""attn_weights: (seq_len, seq_len)"""
import matplotlib.pyplot as plt
import seaborn as sns
plt.figure(figsize=(10, 8))
sns.heatmap(attn_weights.detach().cpu().numpy(),
xticklabels=tokens, yticklabels=tokens,
cmap='viridis')
plt.title('Attention Weights')
plt.show()
5. BERT：双向编码器
5.1 预训练任务
Masked Language Model（MLM）：随机 mask 15% 的 token，预测原词。
Next Sentence Prediction（NSP）：判断两句是否连续（已在 RoBERTa 中证明效果有限，后续版本移除）。
from transformers import BertTokenizer, BertForSequenceClassification
tokenizer = BertTokenizer.from_pretrained('bert-base-chinese')
model = BertForSequenceClassification.from_pretrained(
'bert-base-chinese', num_labels=2)
text = "这是一个测试句子"
inputs = tokenizer(text, return_tensors='pt', padding=True, truncation=True)
outputs = model(**inputs)
logits = outputs.logits
5.2 微调策略
| 任务 | 输出层修改 | 示例 |
|---|---|---|
| 文本分类 | Linear(embed_dim, num_classes) | 情感分析 |
| 命名实体识别 | Linear(embed_dim, num_labels) | 人名/地名提取 |
| 问答 | Start/End 指针 | SQuAD |
| 句子相似度 | Sentence pair → similarity | STS |
5.3 BERT 变体
| 模型 | 改进 | 特点 |
|---|---|---|
| RoBERTa | 更优训练策略 | 移除 NSP，更大 batch |
| ALBERT | 参数共享 + 因子化 | 大幅减少参数量 |
| DeBERTa | 解耦注意力 | 更好的位置处理 |
| ELECTRA | 判别式预训练 | 替换 token 检测 |
6. GPT：生成式预训练 Transformer
6.1 自回归生成
GPT 使用 Decoder-only 架构，通过自回归方式预测下一个 token。
from transformers import GPT2Tokenizer, GPT2LMHeadModel
tokenizer = GPT2Tokenizer.from_pretrained('gpt2')
model = GPT2LMHeadModel.from_pretrained('gpt2')
prompt = "The future of artificial intelligence"
inputs = tokenizer(prompt, return_tensors='pt')
# 生成
outputs = model.generate(
**inputs,
max_length=100,
num_return_sequences=1,
temperature=0.8, # 采样温度
top_k=50, # 限制候选池
top_p=0.95, # 核采样
do_sample=True
)
generated = tokenizer.decode(outputs[0], skip_special_tokens=True)
print(generated)
6.2 采样策略
| 策略 | 公式 | 效果 |
|---|---|---|
| Greedy | argmax | 确定但可能重复 |
| Temperature | $\exp(z_i/T) / \sum \exp(z_j/T)$ | T<1 保守，T>1 多样 |
| Top-k | 只取概率最高的 k 个 | 限制范围 |
| Top-p (Nucleus) | 累加概率达 p 的最小集合 | 动态候选池 |
| Beam Search | 维护 k 条候选序列 | 最优序列，但多样性差 |
7. Vision Transformer (ViT)
将 Transformer 应用于图像，将图像切分为 patch 序列。
class PatchEmbedding(nn.Module):
def __init__(self, img_size=224, patch_size=16, in_channels=3, embed_dim=768):
super().__init__()
self.patch_size = patch_size
self.n_patches = (img_size // patch_size) ** 2
self.proj = nn.Conv2d(in_channels, embed_dim,
kernel_size=patch_size, stride=patch_size)
def forward(self, x):
x = self.proj(x) # (batch, embed_dim, n_patches^(1/2), n_patches^(1/2))
x = x.flatten(2) # (batch, embed_dim, n_patches)
x = x.transpose(1, 2) # (batch, n_patches, embed_dim)
return x
class ViT(nn.Module):
def __init__(self, img_size=224, patch_size=16, in_channels=3,
num_classes=1000, embed_dim=768, num_heads=12,
depth=12, mlp_ratio=4.0):
super().__init__()
self.patch_embed = PatchEmbedding(img_size, patch_size,
in_channels, embed_dim)
self.cls_token = nn.Parameter(torch.zeros(1, 1, embed_dim))
self.pos_embed = nn.Parameter(
torch.zeros(1, self.patch_embed.n_patches + 1, embed_dim))
self.transformer = nn.ModuleList([
TransformerEncoderLayer(embed_dim, num_heads,
int(embed_dim * mlp_ratio))
for _ in range(depth)
])
self.norm = nn.LayerNorm(embed_dim)
self.head = nn.Linear(embed_dim, num_classes)
def forward(self, x):
x = self.patch_embed(x)
cls = self.cls_token.expand(x.shape[0], -1, -1)
x = torch.cat([cls, x], dim=1)
x = x + self.pos_embed
for layer in self.transformer:
x = layer(x)
x = self.norm(x)
return self.head(x[:, 0]) # 只取 CLS token
8. 高效 Transformer 变体
| 变体 | 核心思想 | 复杂度 |
|---|---|---|
| Linear Attention | 核技巧近似 Softmax | $O(n)$ |
| Sparse Attention | 只关注局部/稀疏位置 | $O(n\sqrt{n})$ |
| Linformer | 低秩近似 K/V | $O(n)$ |
| Flash Attention | 分块计算 + IO 感知 | $O(n^2)$，但内存高效 |
| RWKV | RNN + Attention 混合 | $O(n)$，支持并行训练 |
# Flash Attention (PyTorch 2.0+)
import torch.nn.functional as F
# 自动使用优化的 attention 实现
output = F.scaled_dot_product_attention(q, k, v, is_causal=True)
总结
| 架构 | 核心机制 | 优点 | 缺点 | 代表模型 |
|---|---|---|---|---|
| RNN | 循环状态传递 | 顺序建模直观 | 梯度消失、慢 | LSTM Text Gen |
| LSTM | 门控 + 细胞状态 | 解决长依赖 | 仍串行、复杂 | 早期 NLP |
| GRU | 简化门控 | 快、参数量少 | 表达能力略低 | 轻量序列任务 |
| Seq2Seq+Attn | Encoder-Decoder + 对齐 | 可解释 | 计算复杂 | GNMT |
| Transformer | Self-Attention | 并行、全局依赖 | $O(n^2)$ 复杂度 | BERT/GPT |
| ViT | Patch + Transformer | 统一架构 | 需大数据 | 视觉 Transformer |
现代 NLP 的核心是 Transformer Encoder（BERT 系列，用于理解）和 Decoder（GPT 系列，用于生成）。理解 Self-Attention 的 Q/K/V 机制与位置编码，是深入学习所有大模型的基础。
继续阅读
探索更多技术文章
浏览归档，发现更多关于系统设计、工具链和工程实践的内容。