{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "4f9a2c43",
   "metadata": {},
   "source": [
    "# 进阶 · QLoRA 与现代对齐算法\n",
    "\n",
    "## Goal\n",
    "\n",
    "先审计 QLoRA 配置和偏好数据，再手算 DPO 对损失并比较 DPO、ORPO、GRPO 的数据合同；Notebook 默认不启动昂贵 GPU 训练。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7f27f18e",
   "metadata": {},
   "source": [
    "## Setup\n",
    "\n",
    "从实验仓库根目录启动 Jupyter。Colab 用户先克隆自己的仓库、进入 `downloads/llm-priority-labs`，再按 `ENGINEERING_WORKFLOW.md` 安装锁定依赖。下面不会静默修改环境。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "432298db",
   "metadata": {},
   "outputs": [],
   "source": [
    "from pathlib import Path\n",
    "import sys\n",
    "\n",
    "ROOT = Path.cwd()\n",
    "if not (ROOT / \"data\").exists():\n",
    "    ROOT = ROOT.parent\n",
    "if not (ROOT / \"data\").exists():\n",
    "    raise RuntimeError(\"请从 llm-priority-labs 根目录启动 Notebook，或先在 Colab 克隆仓库并进入该目录\")\n",
    "sys.path.insert(0, str(ROOT / \"src\"))\n",
    "IN_COLAB = \"google.colab\" in sys.modules\n",
    "print(\"实验根目录:\", ROOT)\n",
    "print(\"运行环境:\", \"Google Colab\" if IN_COLAB else \"本地 Jupyter\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fe2e7e3c",
   "metadata": {},
   "source": [
    "## Steps\n",
    "\n",
    "### 1. QLoRA 不是一个魔法开关\n",
    "\n",
    "4-bit 基座权重、LoRA 可训练适配器、计算 dtype、目标模块和显存预算必须分别说明。先生成计划，不满足 Linux + NVIDIA GPU 条件时停在 dry-run。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ae875548",
   "metadata": {},
   "outputs": [],
   "source": [
    "from learning_labs.advanced_training import QLoRAPlan, alignment_method_card, dpo_pair_loss, validate_preference_rows\n",
    "\n",
    "plan = QLoRAPlan(\n",
    "    model_id=\"Qwen/Qwen2.5-0.5B-Instruct\",\n",
    "    dataset_path=\"data/preference_tiny.jsonl\",\n",
    ")\n",
    "print(plan.manifest())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8bd123ff",
   "metadata": {},
   "source": [
    "### 2. 在训练前审计偏好数据\n",
    "\n",
    "每行都要有 `prompt/chosen/rejected`。chosen 与 rejected 相同、空文本和重复对都会削弱监督信号。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8c7a80d8",
   "metadata": {},
   "outputs": [],
   "source": [
    "import json\n",
    "\n",
    "rows = [json.loads(line) for line in (ROOT / \"data\" / \"preference_tiny.jsonl\").read_text(encoding=\"utf-8\").splitlines()]\n",
    "audit = validate_preference_rows(rows)\n",
    "print(audit)\n",
    "assert audit[\"identical_pairs\"] == 0"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4f6c4e1a",
   "metadata": {},
   "source": [
    "### 3. 看懂 DPO 的核心比较\n",
    "\n",
    "DPO 比较策略模型相对参考模型对 chosen/rejected 的偏好边际。下面传入的是序列 log-probability，不是普通概率。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f2c3d069",
   "metadata": {},
   "outputs": [],
   "source": [
    "weak = dpo_pair_loss(-8.0, -8.2, -7.0, -7.5, beta=0.1)\n",
    "strong = dpo_pair_loss(-6.0, -8.0, -7.0, -7.5, beta=0.1)\n",
    "print(\"偏好较弱:\", weak, \"偏好较强:\", strong)\n",
    "assert strong < weak"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "afa83c10",
   "metadata": {},
   "source": [
    "### 4. 区分 DPO、ORPO 与 GRPO\n",
    "\n",
    "三者不是同一训练器的三个别名：前两者常用成对偏好，GRPO 需要对同一提示生成一组回答并计算可信奖励。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fe197964",
   "metadata": {},
   "outputs": [],
   "source": [
    "for method in (\"dpo\", \"orpo\", \"grpo\"):\n",
    "    print(alignment_method_card(method))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5f9a2bd3",
   "metadata": {},
   "source": [
    "## Checks"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d159c522",
   "metadata": {},
   "outputs": [],
   "source": [
    "assert plan.quant_type == \"nf4\" and plan.double_quant\n",
    "assert audit[\"rows\"] >= 2\n",
    "print(\"[PASS] 配置、数据合同、目标函数和算法边界已检查\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d8d71563",
   "metadata": {},
   "source": [
    "## Challenge\n",
    "\n",
    "在有权限的 Linux + NVIDIA GPU 环境中执行 `python scripts/advanced_qlora_alignment.py --method sft --run`。先跑 SFT 基线，再分别运行 DPO/ORPO；GRPO 必须先实现可测试奖励。记录峰值显存、训练步数、吞吐和独立验证集，不只贴一条生成样例。\n",
    "\n",
    "## Next Steps\n",
    "\n",
    "先读 `ENGINEERING_WORKFLOW.md` 的进阶环境说明，再运行推理基准 Notebook。"
   ]
  }
 ],
 "metadata": {
  "colab": {
   "name": "进阶 · QLoRA 与现代对齐算法",
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "name": "python",
   "version": "3.11"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}