LLM Notes

LLM

Prompt tuning involves freezing the pre-trained model's parameters and optimizing only a small set of additional parameters, often referred to as "soft prompts." These soft prompts are learned during training and serve as a bridge between the model's pre-trained knowledge and the specific task at hand.

---

2.2 Transformer 基础架构

LLM 依赖于 2017 年 Google 提出的 Transformer 模型，该架构相比传统的 RNN（递归神经网络）和 LSTM（长短时记忆网络）具有更高的训练效率和 更强的长距离依赖建模能力。Transformer 由多个关键组件组成：1. 自注意 力机制（Self-Attention）：模型在处理文本时，会自动关注句子中的重要单 词，理解不同词语间的联系。2. 多头注意力（Multi-Head Attention）：使用 多个注意力头同时分析不同的语义信息，使得模型的理解能力更强。3. 前 馈神经网络（FFN）：非线性变换模块，提升模型的表达能力。4. 位置编码 （Positional Encoding）：在没有循环结构的情况下，帮助模型理解单词的顺 序信息。

Transformer 结构的优势

1. 高效的并行计算：摒弃循环结构，使计算速度大幅提升。
2. 更好的上下文理解：注意力机制可捕捉长文本中的远程依赖关系。
3. 良好的可扩展性：可适配更大规模模型训练，增强 AI 泛化能力。

2.3 LLM 基本训练方法

2.3.1 预训练（Pretraining）

LLM 训练通常采用大规模无监督学习，即：1. 从互联网上收集大量文本数 据，如书籍、新闻、社交媒体等。2. 让模型学习词语之间的概率分布，理解 句子结构。3. 训练目标是最小化预测误差，使其能更好地完成语言任务。

2.3.2 监督微调（Supervised Fine-Tuning, SFT）

在预训练之后，通常需要对模型进行监督微调（SFT）：使用人工标注的数 据集，让模型在特定任务上优化表现。调整参数，使其更符合人类需求，如 问答、对话生成等任务。

2.3.3 强化学习（Reinforcement Learning, RL）

采用强化学习（RL）方法进行优化，主要通过人类反馈强化学习（RLHF, Reinforcement Learning from Human Feedback）：

强化学习（RLHF）优化过程

• 1：人类标注者提供高质量回答。
• 2：模型学习人类评分标准，提高输出质量。
• 3：强化训练，使得生成的文本更符合人类偏好。

RMSNorm

The primary difference between Layer Normalization (LayerNorm) and Root Mean Square Layer Normalization (RMSNorm) lies in whether the "mean" of the activations is subtracted during the normalization process.

While LayerNorm re-centers and re-scales activations, RMSNorm only re-scales them. This small change makes RMSNorm computationally more efficient, which is why it has become the default choice for modern Large Language Models like Llama 3 and Gopher.

---

1. Mathematical Breakdown

Layer Normalization

LayerNorm standardizes the activations $x$ of a layer to have zero mean and unit variance. For a vector $x$ of dimension $d$:

$\bar{x}_i = \frac{x_i - \mu}{\sqrt{\sigma^2 + \epsilon}} \cdot \gamma_i + \beta_i$

Where:

• Mean ($\mu$): $\frac{1}{d} \sum_{j=1}^{d} x_j$
• Variance ($\sigma^2$): $\frac{1}{d} \sum_{j=1}^{d} (x_j - \mu)^2$
• $\gamma$ and $\beta$: Learnable gain and bias parameters.

RMSNorm

RMSNorm simplifies this by assuming that the re-centering (subtracting the mean) is not strictly necessary for the stabilization of deep networks. It only scales by the Root Mean Square:

$\bar{x}_i = \frac{x_i}{\sqrt{\frac{1}{d} \sum_{j=1}^{d} x_j^2 + \epsilon}} \cdot \gamma_i$

Where:

• RMS: $\sqrt{\frac{1}{d} \sum_{j=1}^{d} x_j^2}$
• $\gamma$: A learnable gain parameter (usually, the bias $\beta$ is removed).

---

2. Key Differences

Feature	Layer Normalization (LN)	RMSNorm
Centering	Subtracts the mean (re-centering).	Does not subtract the mean.
Scaling	Scales by standard deviation.	Scales by the RMS of the inputs.
Parameters	Uses both weight ($\gamma$) and bias ($\beta$).	Typically uses only weight ($\gamma$).
Computation	Slower (requires mean and variance).	Faster (approx. 10–40% gain in norm op).
Invariance	Re-scaling and Re-centering invariant.	Only Re-scaling invariant.

---

3. Why is RMSNorm winning?

1. Efficiency: In the context of LLMs with billions of parameters, calculating the mean and subtracting it across every layer adds up. RMSNorm reduces the number of operations per normalization step.
2. Performance: Empirical evidence (notably from the original RMSNorm paper by Zhang and Sennrich) suggests that removing the mean-centering does not hurt the convergence or the final accuracy of the model.
3. Numerical Stability: It provides similar benefits to LayerNorm in preventing exploding or vanishing gradients by keeping the activation magnitudes in check.

---

Comparison in Code (PyTorch-style logic)

# LayerNorm
mean = x.mean(-1, keepdim=True)
var = x.var(-1, keepdim=True, unbiased=False)
x_norm = (x - mean) / torch.sqrt(var + eps)

# RMSNorm
rms = torch.sqrt(torch.mean(x**2, dim=-1, keepdim=True) + eps)
x_norm = x / rms