计算机视觉让机器"看懂"世界，涵盖目标检测、图像分割、文字识别与图像生成。本文深入讲解 YOLO 系列演进、实例分割、OCR 全栈方案与生成对抗网络。
1. 目标检测：YOLO 系列
1.1 YOLOv8 使用
YOLO (You Only Look Once) 是实时目标检测的代表，将检测视为回归问题。
from ultralytics import YOLO
# 加载预训练模型
model = YOLO('yolov8n.pt') # nano 版本，轻量快速
# 目标检测
results = model('image.jpg')
for r in results:
boxes = r.boxes # 边界框
for box in boxes:
cls = int(box.cls) # 类别
conf = float(box.conf) # 置信度
xyxy = box.xyxy[0].tolist() # [x1, y1, x2, y2]
print(f"类别: {model.names[cls]}, 置信度: {conf:.3f}, 位置: {xyxy}")
# 训练自定义数据集
model = YOLO('yolov8n.pt')
model.train(data='custom_data.yaml', epochs=100, imgsz=640)
# 导出 ONNX
model.export(format='onnx')
1.2 YOLO 系列演进
| 版本 | 年份 | 核心改进 | mAP | FPS |
|---|---|---|---|---|
| YOLOv1 | 2016 | 单阶段检测开创 | 63.4 | 45 |
| YOLOv3 | 2018 | 多尺度预测、Darknet-53 | 57.9 | 45 |
| YOLOv5 | 2020 | PyTorch 实现、AutoAnchor | 50.7 | 140 |
| YOLOv8 | 2023 | Anchor-Free、解耦头 | 53.9 | 150 |
| YOLOv10 | 2024 | NMS-Free、一致性分配 | 54.4 | 200+ |
1.3 检测输出格式
# COCO 格式
{
"image_id": 1,
"category_id": 1, # person=1
"bbox": [x, y, width, height], # 左上角 + 宽高
"score": 0.95,
"area": width * height
}
# Pascal VOC 格式
# xmin, ymin, xmax, ymax
1.4 评估指标
| 指标 | 定义 |
|---|---|
| IoU | 交并比 = 预测框 ∩ 真实框 / 预测框 ∪ 真实框 |
| mAP@0.5 | IoU=0.5 时的平均精度 |
| mAP@0.5:0.95 | IoU 从 0.5 到 0.95 步进 0.05 的平均 |
| AP50/AP75 | IoU=0.5/0.75 时的 AP |
from ultralytics.utils.metrics import bbox_iou
iou = bbox_iou(pred_box, true_box, xywh=True, CIoU=True)
2. 图像分割
2.1 语义分割
为每个像素分配类别标签。
from torchvision.models.segmentation import fcn_resnet50, deeplabv3_resnet50
# DeepLabV3+
model = deeplabv3_resnet50(pretrained=True)
model.eval()
# 输入图像
from torchvision import transforms
transform = transforms.Compose([
transforms.ToTensor(),
transforms.Normalize(mean=[0.485, 0.456, 0.406],
std=[0.229, 0.224, 0.225])
])
input_tensor = transform(image).unsqueeze(0)
output = model(input_tensor)['out'] # (1, 21, H, W)
predictions = output.argmax(1) # 每个像素的类别
2.2 实例分割
区分同类别的不同实例。
from torchvision.models.detection import maskrcnn_resnet50_fpn
model = maskrcnn_resnet50_fpn(pretrained=True)
model.eval()
predictions = model([transform(image)])
# predictions[0]['boxes']: 边界框
# predictions[0]['labels']: 类别
# predictions[0]['scores']: 置信度
# predictions[0]['masks']: 分割掩膜 (N, 1, H, W)
2.3 U-Net 详解
import torch.nn as nn
class UNet(nn.Module):
def __init__(self, in_channels=3, out_channels=1):
super().__init__()
# 编码器
self.enc1 = self.conv_block(in_channels, 64)
self.enc2 = self.conv_block(64, 128)
self.enc3 = self.conv_block(128, 256)
self.enc4 = self.conv_block(256, 512)
self.pool = nn.MaxPool2d(2)
# 瓶颈
self.bottleneck = self.conv_block(512, 1024)
# 解码器
self.up4 = nn.ConvTranspose2d(1024, 512, 2, stride=2)
self.dec4 = self.conv_block(1024, 512)
self.up3 = nn.ConvTranspose2d(512, 256, 2, stride=2)
self.dec3 = self.conv_block(512, 256)
self.up2 = nn.ConvTranspose2d(256, 128, 2, stride=2)
self.dec2 = self.conv_block(256, 128)
self.up1 = nn.ConvTranspose2d(128, 64, 2, stride=2)
self.dec1 = self.conv_block(128, 64)
self.final = nn.Conv2d(64, out_channels, 1)
def conv_block(self, in_ch, out_ch):
return nn.Sequential(
nn.Conv2d(in_ch, out_ch, 3, padding=1),
nn.BatchNorm2d(out_ch),
nn.ReLU(inplace=True),
nn.Conv2d(out_ch, out_ch, 3, padding=1),
nn.BatchNorm2d(out_ch),
nn.ReLU(inplace=True)
)
def forward(self, x):
# 编码
e1 = self.enc1(x)
e2 = self.enc2(self.pool(e1))
e3 = self.enc3(self.pool(e2))
e4 = self.enc4(self.pool(e3))
# 瓶颈
b = self.bottleneck(self.pool(e4))
# 解码 + Skip Connection
d4 = self.up4(b)
d4 = torch.cat([d4, e4], dim=1)
d4 = self.dec4(d4)
d3 = self.up3(d4)
d3 = torch.cat([d3, e3], dim=1)
d3 = self.dec3(d3)
d2 = self.up2(d3)
d2 = torch.cat([d2, e2], dim=1)
d2 = self.dec2(d2)
d1 = self.up1(d2)
d1 = torch.cat([d1, e1], dim=1)
d1 = self.dec1(d1)
return self.final(d1)
3. OCR 文字识别
3.1 OCR Pipeline
图像 → 文字检测 → 文字识别 → 后处理 → 结构化输出
↑_____ PaddleOCR/EasyOCR ____↑
from paddleocr import PaddleOCR
# 初始化
ocr = PaddleOCR(use_angle_cls=True, lang='ch',
det_model_dir='./models/det',
rec_model_dir='./models/rec')
# 识别
result = ocr.ocr('document.jpg', cls=True)
for line in result[0]:
box = line[0] # 文字框坐标
text = line[1][0] # 识别文本
confidence = line[1][1]
print(f"文本: {text}, 置信度: {confidence:.3f}")
3.2 EasyOCR
import easyocr
reader = easyocr.Reader(['ch_sim', 'en'])
results = reader.readtext('image.jpg')
for (bbox, text, conf) in results:
print(f"{text} ({conf:.2f})")
3.3 自定义 OCR 训练
# 使用 PaddleOCR 训练自定义数据集
# 数据格式：
# image_path\t[{transcription: "文字", points: [[x1,y1],...]}]
# 检测模型训练
!python tools/train.py -c configs/det/ch_PP-OCRv4/ch_PP-OCRv4_det_student.yml
# 识别模型训练
!python tools/train.py -c configs/rec/PP-OCRv4/ch_PP-OCRv4_rec.yml
4. 生成模型
4.1 GAN：生成对抗网络
生成器 G 和判别器 D 的对抗博弈：
$$\min_G \max_D V(D, G) = E[\log D(x)] + E[\log(1 - D(G(z)))]$$
import torch.nn as nn
# 生成器
class Generator(nn.Module):
def __init__(self, latent_dim=100, img_size=28):
super().__init__()
self.model = nn.Sequential(
nn.Linear(latent_dim, 256),
nn.LeakyReLU(0.2),
nn.BatchNorm1d(256),
nn.Linear(256, 512),
nn.LeakyReLU(0.2),
nn.BatchNorm1d(512),
nn.Linear(512, 1024),
nn.LeakyReLU(0.2),
nn.BatchNorm1d(1024),
nn.Linear(1024, img_size * img_size),
nn.Tanh()
)
def forward(self, z):
img = self.model(z)
return img.view(img.size(0), 1, 28, 28)
# 判别器
class Discriminator(nn.Module):
def __init__(self, img_size=28):
super().__init__()
self.model = nn.Sequential(
nn.Linear(img_size * img_size, 512),
nn.LeakyReLU(0.2),
nn.Linear(512, 256),
nn.LeakyReLU(0.2),
nn.Linear(256, 1),
nn.Sigmoid()
)
def forward(self, img):
flat = img.view(img.size(0), -1)
return self.model(flat)
# 训练循环
def train_gan(generator, discriminator, dataloader, epochs=50):
g_optimizer = torch.optim.Adam(generator.parameters(), lr=0.0002)
d_optimizer = torch.optim.Adam(discriminator.parameters(), lr=0.0002)
criterion = nn.BCELoss()
for epoch in range(epochs):
for real_imgs, _ in dataloader:
batch_size = real_imgs.size(0)
# 标签
real = torch.ones(batch_size, 1)
fake = torch.zeros(batch_size, 1)
# 训练判别器
d_optimizer.zero_grad()
real_loss = criterion(discriminator(real_imgs), real)
z = torch.randn(batch_size, 100)
fake_imgs = generator(z)
fake_loss = criterion(discriminator(fake_imgs.detach()), fake)
d_loss = (real_loss + fake_loss) / 2
d_loss.backward()
d_optimizer.step()
# 训练生成器
g_optimizer.zero_grad()
g_loss = criterion(discriminator(fake_imgs), real)
g_loss.backward()
g_optimizer.step()
4.2 条件 GAN (CGAN)
# 将类别标签作为输入条件
class ConditionalGenerator(nn.Module):
def __init__(self, latent_dim, num_classes, img_size):
super().__init__()
self.label_emb = nn.Embedding(num_classes, num_classes)
self.model = nn.Sequential(
nn.Linear(latent_dim + num_classes, 256),
# ...
)
def forward(self, z, labels):
label_input = self.label_emb(labels)
x = torch.cat([z, label_input], dim=1)
return self.model(x)
4.3 Diffusion Models
扩散模型通过逐步去噪从随机噪声生成图像，是当前最高质量的生成方法。
from diffusers import StableDiffusionPipeline
import torch
# 加载 Stable Diffusion
pipe = StableDiffusionPipeline.from_pretrained(
"runwayml/stable-diffusion-v1-5",
torch_dtype=torch.float16
).to("cuda")
# 文生图
prompt = "A beautiful sunset over mountains, digital art"
image = pipe(prompt, num_inference_steps=50, guidance_scale=7.5).images[0]
image.save("output.png")
# 图生图
from diffusers import StableDiffusionImg2ImgPipeline
pipe_img2img = StableDiffusionImg2ImgPipeline.from_pretrained(
"runwayml/stable-diffusion-v1-5",
torch_dtype=torch.float16
).to("cuda")
init_image = Image.open("input.jpg").convert("RGB").resize((512, 512))
image = pipe_img2img(prompt="turn this into anime style",
image=init_image, strength=0.7).images[0]
4.4 扩散模型原理
前向扩散：逐步向图像添加高斯噪声
$$q(x_t | x_{t-1}) = \mathcal{N}(x_t; \sqrt{1-\beta_t}x_{t-1}, \beta_t I)$$
反向去噪：学习神经网络预测噪声，逐步去噪
$$p_\theta(x_{t-1}|x_t) = \mathcal{N}(x_{t-1}; \mu_\theta(x_t, t), \Sigma_\theta(x_t, t))$$
5. 视频理解
5.1 视频动作识别
from torchvision.models.video import r3d_18, mc3_18
model = r3d_18(pretrained=True)
# 输入: (batch, channels, frames, height, width)
# r3d_18: (N, 3, 16, 112, 112)
5.2 目标跟踪
from ultralytics import YOLO
model = YOLO('yolov8n.pt')
results = model.track(source='video.mp4', show=True, tracker="bytetrack.yaml")
6. 模型部署优化
| 优化方案 | 方法 | 加速比 |
|---|---|---|
| 量化 | INT8/FP16 | 2-4x |
| 剪枝 | 移除不重要的权重 | 2-10x |
| 知识蒸馏 | 大模型 → 小模型 | 保持精度 |
| TensorRT | NVIDIA 专用优化 | 5-10x |
| ONNX Runtime | 跨平台推理 | 2-3x |
# TensorRT 导出
import torch
from torch2trt import torch2trt
model = YOLO('yolov8n.pt').model
x = torch.ones((1, 3, 640, 640)).cuda()
model_trt = torch2trt(model, [x])
torch.save(model_trt.state_dict(), "yolov8n_trt.pth")
总结
计算机视觉任务速查：
| 任务 | 推荐方案 | 关键模型 |
|---|---|---|
| 目标检测 | YOLOv8 / RT-DETR | YOLOv8n/s/m/l/x |
| 语义分割 | DeepLabV3+ / U-Net | ResNet + ASPP |
| 实例分割 | Mask R-CNN | Faster R-CNN + Mask Head |
| OCR | PaddleOCR / EasyOCR | PP-OCRv4 |
| 图像生成 | Stable Diffusion | U-Net + CLIP |
| 视频分析 | SlowFast / X3D | 3D CNN |
工程实践：
- 检测任务优先考虑实时性，选 YOLO；高精度需求选两阶段检测器
- 分割任务 U-Net 是基线，DeepLabV3+ 是 SOTA 选择
- OCR 直接使用 PaddleOCR，支持中英文且可自定义训练
- 生成模型 Stable Diffusion 是开源首选，ControlNet 提供可控生成
- 部署时用 TensorRT/OpenVINO 加速，INT8 量化减少显存占用
继续阅读
探索更多技术文章
浏览归档，发现更多关于系统设计、工具链和工程实践的内容。