大模型微调实战：LoRA、QLoRA 与指令数据构造全流程

模型权重      7B x 2B  = 14 GB
梯度          7B x 2B  = 14 GB
Adam 状态     7B x 8B  = 56 GB   ← 大头
激活值与临时  取决于 batch 与序列长度
合计          约 84 GB 起

原始前向：  h = W x
LoRA 前向：  h = W x + (alpha / r) * B A x

W 形状 (d, k) 冻结；A 形状 (r, k) 高斯初始化；
B 形状 (d, r) 零初始化；r << min(d, k)，常取 8 / 16 / 32

target_modules = ["q_proj", "k_proj", "v_proj", "o_proj",
                  "gate_proj", "up_proj", "down_proj"]

instruction : 用户要模型做什么（任务描述）
input       : 任务需要的输入（可为空）
output      : 期望的模型回答（监督信号）

sample = {
    "messages": [
        {"role": "system",    "content": "你是一名严谨的法律助手"},
        {"role": "user",      "content": "帮我解释这条合同的免责条款"},
        {"role": "assistant", "content": "该条款的含义是……"},
    ]
}

from transformers import AutoTokenizer

tok = AutoTokenizer.from_pretrained("Qwen/Qwen2.5-7B-Instruct")
text = tok.apply_chat_template(
    sample["messages"], tokenize=False,
    add_generation_prompt=False,      # 训练时不要生成提示
)
print(text)

def mask_prompt_labels(labels, prompt_len):
    labels = list(labels)
    for i in range(min(prompt_len, len(labels))):
        labels[i] = -100
    return labels

4bit 基座权重      约 3.5 GB
LoRA 参数与梯度    约 0.2 GB
优化器状态         约 0.4 GB
激活值（checkpoint）约 6~10 GB
CUDA 上下文与碎片  约 1 GB
峰值               约 12~16 GB   → 24GB 卡可跑

from transformers import TrainingArguments

args = TrainingArguments(
    output_dir="./out",
    per_device_train_batch_size=1,
    gradient_accumulation_steps=16,   # 等效 batch 16
    gradient_checkpointing=True,      # 用计算换显存
    optim="paged_adamw_8bit",         # 优化器状态 8bit
    learning_rate=2e-4,
    lr_scheduler_type="cosine",
    warmup_ratio=0.03,
    num_train_epochs=3,
    bf16=True,                        # 优先 bf16 而非 fp16
)

import torch
from transformers import AutoModelForCausalLM, BitsAndBytesConfig

bnb = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_use_double_quant=True,
    bnb_4bit_compute_dtype=torch.bfloat16,
)
model = AutoModelForCausalLM.from_pretrained(
    "Qwen/Qwen2.5-7B-Instruct",
    quantization_config=bnb,
    device_map="auto",
)

from peft import LoraConfig, get_peft_model, prepare_model_for_kbit_training

model = prepare_model_for_kbit_training(model)   # 让 4bit 模型可训练
lora_cfg = LoraConfig(
    r=16, lora_alpha=32, lora_dropout=0.05,
    target_modules=["q_proj", "k_proj", "v_proj", "o_proj"],
    task_type="CAUSAL_LM",
)
model = get_peft_model(model, lora_cfg)
model.print_trainable_parameters()
# trainable params: 约 8~20M || all params: 7B || trainable%: 0.2%

from transformers import Trainer

trainer = Trainer(model=model, args=args, train_dataset=ds, data_collator=collator)
trainer.train()
model.save_pretrained("./lora-adapter")     # 只存适配器，几十 MB

from peft import PeftModel
from transformers import AutoModelForCausalLM

base = AutoModelForCausalLM.from_pretrained(
    "Qwen/Qwen2.5-7B-Instruct", torch_dtype=torch.bfloat16)
merged = PeftModel.from_pretrained(base, "./lora-adapter").merge_and_unload()
merged.save_pretrained("./merged-7b")

from vllm import LLM, SamplingParams

llm = LLM(model="./merged-7b", dtype="bfloat16", max_model_len=4096)
out = llm.generate(["解释一下什么是低秩分解"],
                   SamplingParams(temperature=0.7, max_tokens=256))
print(out[0].outputs[0].text)

import json, re

def structural_pass_rate(preds):
    hits = 0
    for p in preds:
        m = re.search(r"\{.*\}", p, re.S)
        try:
            json.loads(m.group(0)); hits += 1
        except (json.JSONDecodeError, AttributeError):
            pass
    return hits / len(preds)

选基座 → 构造指令数据 → 套 chat template → 4bit 加载 + LoRA 注入
   → 训练（小 lr、2~3 epoch、梯度检查点）→ 合并或热插拔
   → 三层评测 → 混合通用数据防遗忘 → 上线监控