1. Introduction

Update (2026-10-28): This revision expands the optimizers and learning rate strategies in §5, and updates FP8 / FlashAttention-4 in §9 and reasoning reinforcement learning and hybrid attention architectures in §11. Years generally refer to first public release; conference publication years are noted separately. Performance claims are limited to the experimental settings of the original papers. Primary sources for the additions are listed at the end.

Deep learning (Deep Learning) is a machine learning method with multi-layer neural network as the core. AlexNet’s breakthrough in the ImageNet Challenge in 2012 accelerated the popularity of deep learning in fields such as computer vision, natural language processing, speech recognition, and scientific computing. Representative systems such as GPT, BERT, Stable Diffusion, and AlphaFold are also built on deep neural networks.

The power of deep learning lies in its ability to automatically learn multi-level abstract feature representations from raw data (pixels, words, signals) without the need for manual feature engineering. However, training a high-performance deep network is not easy - problems such as vanishing gradients, overfitting, and learning rate adjustment have long troubled researchers, which has given rise to a set of systematic training techniques.

To understand deep learning, you need to grasp two levels at the same time: architecture design (how to build the network) and training method (how to effectively optimize the network). Both are indispensable. If you only know how to stack network layers like building blocks without understanding what problem each training technique solves, you will often fall into the dilemma of “adding Dropout will only make it worse” or “replacing Adam will not improve anything”.

This article aims to systematically sort out the core principles and key technologies of deep learning, and provide a searchable map for learning, reproduction, and research. The full text is developed according to the following context:

• §2–3 Basics: machine learning three-step framework, neurons and backpropagation
• §4 Architecture: The main line of evolution of MLP → CNN → RNN/LSTM → Transformer, and supplemented by graph neural network (GNN)
• §5 Training optimization: From AdamW to Muon / SOAP / NorMuon; learning rate scheduling and Schedule-Free; initialization, normalization, and residual connections
• §6 Loss and regularization: Selection of various types of Loss, Dropout, data augmentation, AdamW
• §7 Method classification summary: Map all techniques to the “three steps × two goals” coordinate system
• §8 Benchmark: MNIST/CIFAR/ImageNet/GLUE/MMLU/HumanEval/Long Context and other classic reviews
• §9 Efficient training and inference: BF16 / FP8, gradient clipping/checkpointing, and FlashAttention 1–4
• §10 Model compression and adaptation: MoE, LoRA/PEFT, quantification, knowledge distillation and other deployment-oriented lightweight methods
• §11 Frontier architectures and large models: RLHF / GRPO / RLVR, generative models (AE/VAE/GAN), diffusion models, Mamba/SSMs, and hybrid attention

Reading Suggestion: Beginners can read in the order of §2–§6; when training the model, give priority to §5–§7 and §9; when focusing on large models, focus on reading §4.3, §10 and §11. The accuracy, parameter count and year in the article are used to illustrate the historical context and are affected by the model version, data processing and evaluation protocol. When reproducing the experiment, the original paper or official report should prevail.

2. Overview of deep learning basics

2.1 What is deep learning?

Deep learning is a branch of machine learning, using Artificial Neural Network (ANN) as the foundation model. “Deep” usually means that the model contains multiple layers of nonlinear transformations; it does not have a fixed layer threshold that applies to all tasks. Stacking multiple layers enables the network to extract increasingly abstract features layer by layer.

The basic computing unit of a neural network is neuron (Neuron):

Here, $x_i$ is the input, $w_i$ is the weight, $b$ is the bias, and $f$ is the activation function. Multiple neurons are stacked into layers, and multiple layers are connected in series to form a deep network, which ultimately forms a complex mapping function from input to output.

Schematic diagram of

Universal approximation theorem

One of the important theoretical basis for the expressive ability of neural network is universal approximation theorem (Cybenko 1989, Hornik 1991):

When conditions such as continuity and non-polynomial activation functions are met, a single hidden layer feedforward network can approximate any continuous function with arbitrary accuracy on a compact set as long as the width is sufficient.

This conclusion answers the fundamental question of “Why can neural networks work?”, but there are also two key limitations that must be clarified:

The advantage of depth : Theoretically (Telgarsky 2016 ) There is a class of functions that deep networks can express with far fewer units than shallow networks. It should be noted that the theorem describes the expression ability and parameter efficiency. It does not guarantee that the optimization process will find these parameters, nor does it directly guarantee the generalization performance.

Why deep learning took off

2.2 Machine learning three-step framework

Any machine learning method can be broken down into three steps. Understanding this framework is the basis for all subsequent training techniques:

1. Step 1 - Define Loss Function (loss function): Measure the difference between the model output and the correct answer, that is, “how to judge good or bad”
2. Step 2 - Determine the function search space (Model Architecture): Select the network structure and delineate the scope of candidate functions, that is, “where to search”
3. Step 3 - Optimization: Find the optimal function that minimizes Loss in the search space, that is, “how to search efficiently”

The ultimate goal of training is to find a function whose Loss is low on the training set (Training Set) and remains low on the validation set (Validation Set). Whether the training error can be reduced mainly reflects Optimization, and the gap between training error and verification error reflects Generalization; the two will affect each other in practice.

2.3 Two core goals

Deep learning training techniques are divided into two categories according to the problems they solve:

Judgment Principle: First look at the training Loss, and then look at the gap between the training and validation curves. When the training loss cannot be reduced, check the learning rate, initialization, normalization and data pipeline first; when the training loss is already low but the validation loss is significantly higher, then consider regularization and data augmentation. Actual problems may involve both types of factors.

2.4 Development Timeline

flowchart LR
    subgraph G1990 ["1990s laying foundation"]
        A["LeNet 1998<br/>CNN laying foundation"] --> B["Backpropagation<br/>Popularize"]
    end
    subgraph G2012 ["2012-2014 break out"]
        C["AlexNet 2012<br/>ImageNet breakthrough"] --> D["Dropout 2014"]
        D --> E["Adam 2014"]
    end
    subgraph G2015 ["2015-2017 deepen"]
        F["ResNet 2015<br/>Skip Connection"] --> G["BN 2015"]
        G --> H["AdamW 2017"]
    end
    subgraph G2018 ["2018-2022 pre-training"]
        I["BERT/GPT 2018<br/>pre-training paradigm"] --> J["Transformer<br/>Dominate"]
        J --> R["RoPE 2021<br/>Rotational position encoding"]
        R --> K["LoRA 2021<br/>Efficient fine-tuning of parameters"]
    end
    subgraph G2022 ["2022 so far large model"]
        L["Chinchilla<br/>2022 Scaling Laws"] --> M["ChatGPT 2022<br/>RLHF Alignment"]
        M --> N["Flash Attn 2/3<br/>2023-2024"]
        N --> O["Muon / SOAP<br/>2024 matrix optimization"]
        O --> P["GRPO / reasoning RL<br/>2024-2025"]
        P --> Q["FlashAttention-4<br/>2026 hardware co-design"]
    end
    B --> C
    E --> F
    H --> I
    K --> L

2.5 Main abbreviation

• ANN: Artificial Neural Network
• MLP: Multi-Layer Perceptron
• CNN: Convolutional Neural Network
• RNN: Recurrent Neural Network
• LSTM: Long Short-Term Memory (long short-term memory network)
• SGD: Stochastic Gradient Descent
• BN: Batch Normalization
• LN: Layer Normalization
• LR: Learning Rate
• PEFT: Parameter-Efficient Fine-Tuning
• SFT: Supervised Fine-Tuning
• RLHF: Reinforcement Learning from Human Feedback (reinforcement learning from human feedback)

3. Neural network basics

3.1 Multi-layer perceptron

Multi-Layer Perceptron (MLP) is the most basic form of neural network. It consists of an input layer, several hidden layers (Hidden Layer) and an output layer. Each layer is fully connected (Fully Connected):

Fully connected means that each neuron in layer $l$ is connected to all neurons in layer $l-1$, and the parameter count is the product of the number of neurons in the two layers. When the input dimension is extremely large (for example, the color image of 1000×1000 has 3 million dimensions after expansion), the number of fully connected parameters will exceed billions, which is not only difficult to train, but also extremely easy to overfitting. Specialized architectures such as CNN were proposed to solve this problem.

3.2 Activation function

Activation functions introduce nonlinearity to neural networks, allowing them to fit complex functions. Without nonlinear activation, a multi-layer network is equivalent to a single-layer linear model.

The success of ReLU (Rectified Linear Unit) lies in the fact that the gradient in the positive area is always 1, which effectively alleviates the vanishing gradient problem of deep networks and enables stable training of networks with dozens or even hundreds of layers.

3.3 Backpropagation algorithm

Backpropagation is the core algorithm for training neural networks. Based on the chain rule, the gradient of Loss for each parameter is transferred from the output layer back to the input layer layer by layer:

flowchart LR
    Loss["Loss<br/>Calculate losses"] -->|"∂L/∂ŷ"| OutLayer["output layer<br/>Calculate gradient"]
    OutLayer -->|"∂L/∂h²"| H2["Hidden layer 2<br/>Calculate gradient"]
    H2 -->|"∂L/∂h¹"| H1["Hidden layer 1<br/>Calculate gradient"]
    H1 -->|"∂L/∂W¹"| Update["Parameter update<br/>gradient descent"]
    style Loss fill:#ff9999
    style Update fill:#99ff99

The Computation Graph automatically records the forward calculation path, and during back propagation the gradient is transferred along the path in the opposite direction. PyTorch’s autograd mechanism is based on this principle. Users only need to define forward calculations, and the framework automatically completes gradient calculations.

A tiny backprop example

Consider the simplest single hidden layer network: $y = w_2 \cdot \sigma(w_1 \cdot x)$, with loss $\mathcal{L} = \frac{1}{2}(y - t)^2$. Set $x=1, w_1=0.5, w_2=0.8, t=0.5$, activate $\sigma$ and take Sigmoid.

forward:

reverse (chain rule stepwise backtracking):

observes: the gradient $\partial \mathcal{L}/\partial w_2 \approx 10^{-3}$ near the output, the gradient $\partial \mathcal{L}/\partial w_1 \approx 10^{-4}$ near the input - even though there are only 2 layers, the gradients are attenuated by $\sim 3$ times. Every additional layer in the deep network requires one more multiplication of $\sigma’(\cdot) \leq 0.25$. This is the source of the vanishing gradient.

Three variations of gradient descent

There are three forms of gradient descent depending on the number of samples used per update:

Modern deep learning almost all uses Mini-batch SGD. The “SGD” spoken in the industry usually refers to the mini-batch version. The choice of batch size directly affects training stability, hardware utilization and final generalization - large batches converge quickly but generalization is slightly worse (Keskar et al., 2017 “On Large-Batch Training”), which is an important parameter adjustment dimension.

4. Classic network architecture

4.1 Convolutional Neural Network

Convolutional Neural Network (CNN) is a standard architecture for processing grid-shaped data (images, time series signals). CNN imposes two key constraints on MLP:

Receptive Field (receptive field): Each neuron only observes a local area of the input (such as the kernel of 3×3), rather than the entire image. Local patterns (edges, textures) in images can be detected using only local perception, without the need for a global view.

Parameter Sharing (parameter sharing): Similar neurons in different positions share the same set of parameters (filter/convolution kernel). The same pattern (such as a horizontal edge) appearing anywhere in the image should be processed by the same detector - this constraint brings translation equivariance (Translation Equivariance); only after pooling or global aggregation, the model will obtain a certain translation invariance.

Convolution hyperparameters

output space size formula:

Regular convolution ($d=1$) simplifies to $\text{out} = \lfloor (\text{in} + 2p - k)/s \rfloor + 1$.

Special convolution

Pooling layer (Pooling)

Pooling is done on the feature map Downsampling , reduce the spatial size, expand the receptive field, and provide certain translation invariance. Pooling layer No parameters to learn .

The meaning of GAP: The fully connected layer of VGG-16 occupies the 123M parameters (accounting for the total parameters 89%); this part can be omitted directly after switching to GAP. This is because the parameters of ResNet and EfficientNet are much smaller than those of VGG one of the key reasons.

A typical CNN consists of alternating convolutional layers (extracting features) and pooling layers (downsampling), finally followed by a fully connected layer or GAP output prediction. The following is the structure of LeNet-5 (LeCun, 1998), the foundational work of CNN:

CNN architecture evolution comparison

Key Insight: ResNet-50 (25M parameters) outperforms VGG-16 (138M parameters) with only 18%; EfficientNet-B0 (5.3M) reaches the same accuracy as VGG, and the number of parameters is only 4%. More parameters ≠ higher performance, architectural design is crucial.

ResNet residual block structure

4.2 Recurrent Neural Network and LSTM

Recurrent Neural Network (RNN) is specially designed to process sequence data and passes information between time steps through the hidden state $h_t$:

RNN uses the same set of parameters ** $W_h, W_x$ at each time step, making it naturally adaptable to variable-length sequences. The following figure shows the “unfolding” of RNN in the time dimension:

flowchart LR
    x0["x₀"] --> h0["h₀"]
    x1["x₁"] --> h1["h₁"]
    x2["x₂"] --> h2["h₂"]
    x3["x₃"] --> h3["h₃"]
    h0 -->|"Wₕ"| h1
    h1 -->|"Wₕ"| h2
    h2 -->|"Wₕ"| h3
    h0 --> y0["y₀"]
    h1 --> y1["y₁"]
    h2 --> y2["y₂"]
    h3 --> y3["y₃"]
    style h0 fill:#e1f5ff
    style h1 fill:#e1f5ff
    style h2 fill:#e1f5ff
    style h3 fill:#e1f5ff

However, standard RNN faces serious long-range dependencies problem: when the gradient is backpropagated in the time dimension, the matrix $W_h$ needs to be multiplied $T$ times. When the maximum singular value of $W_h$ $< 1$, gradients decay exponentially with sequence length (vanishing gradients), $> 1$ exponentially explodes (exploding gradients), making it difficult for the model to remember distant contexts.

LSTM (Long Short-Term Memory, Hochreiter & Schmidhuber, 1997) solves this problem by introducing gating mechanism (Gating Mechanism):

LSTM maintains two states: hidden state $h_t$ (short-term memory) and cell state $c_t$ (long-term memory).

Internal data flow of a single LSTM cell:

flowchart LR
    subgraph INPUT ["input"]
        XT["xₜ"]
        HT1["hₜ₋₁"]
        CT1["cₜ₋₁"]
    end
    CONCAT["Splicing [hₜ₋₁, xₜ]"]
    XT --> CONCAT
    HT1 --> CONCAT
    CONCAT --> FGATE["oblivion door fₜ<br/>σ(W·)"]
    CONCAT --> IGATE["input gate iₜ<br/>σ(W·)"]
    CONCAT --> CAND["candidate c̃ₜ<br/>tanh(W·)"]
    CONCAT --> OGATE["output gate oₜ<br/>σ(W·)"]
    CT1 --> MUL1["× fₜ<br/>forget old memories"]
    FGATE --> MUL1
    IGATE --> MUL2["× c̃ₜ<br/>write new memory"]
    CAND --> MUL2
    MUL1 --> ADD["＋"]
    MUL2 --> ADD
    ADD --> CT["cₜ<br/>new cell state"]
    CT --> TANH["tanh"]
    TANH --> MUL3["× oₜ"]
    OGATE --> MUL3
    MUL3 --> HT["hₜ<br/>New hidden state"]
    style CT fill:#ffe1c4
    style HT fill:#c4e1ff
    style CT1 fill:#ffe1c4
    style HT1 fill:#c4e1ff

cell status update formula:

Why can gating alleviate vanishing gradients? Cell state update $c_t = f_t \odot c_{t-1} + \ldots$ is a additive path (compare to RNN’s multiplicative recursion $h_t = f(Wh_{t-1} + \ldots)$). When forgetting gate $f_t \approx 1$, $c_{t-1}$ is passed to $c_t$ with almost no attenuation, and the gradient can remain stable when back propagating along the cell state - similar to ResNet’s skip connection, but occurring in the time dimension rather than the depth dimension.

GRU: a simplified version of LSTM

Gated Recurrent Unit (GRU, Cho et al., 2014) simplifies the 3 gates of LSTM into 2 gates, and merges the cell state and hidden state:

Empirical conclusion: Chung et al., 2014 System experiments show that GRU and LSTM have equivalent performance on most tasks. GRU has fewer parameters and faster training, and is more commonly used in short and medium sequence tasks (conversation, emotion classification); LSTM still has advantages in tasks that require strict long-term memory (machine translation of long sentences, speech recognition).

Bidirectional RNN

The standard RNN can only utilize the information of past (from left to right). But for many tasks (named entity recognition, sentiment analysis, reading comprehension), the context of and future is equally critical - for example, to determine whether “bank” is “bank” or “river bank”, you need to look at the following words.

Bidirectional RNN (Schuster & Paliwal, 1997) runs RNN in two directions at the same time, output splicing:

flowchart LR
    x1["x₁"] --> fh1["→h₁"]
    x2["x₂"] --> fh2["→h₂"]
    x3["x₃"] --> fh3["→h₃"]
    fh1 --> fh2 --> fh3
    x1 --> bh1["←h₁"]
    x2 --> bh2["←h₂"]
    x3 --> bh3["←h₃"]
    bh3 --> bh2 --> bh1
    fh1 --> c1["[→;←] h₁"]
    bh1 --> c1
    fh2 --> c2["[→;←] h₂"]
    bh2 --> c2
    fh3 --> c3["[→;←] h₃"]
    bh3 --> c3
    style c1 fill:#c4e1ff
    style c2 fill:#c4e1ff
    style c3 fill:#c4e1ff

Limitations: Bidirectional RNN must see the entire sequence to calculate, cannot be used for autoregressive generation tasks (no future information when generating the next word). Therefore:

• is suitable for: sequence annotation, classification, NER, and reading comprehension (BiLSTM-CRF was the mainstream solution for NER in the years 2016–2018)
• is not suitable for: Decoder side of language model and machine translation

BERT’s MLM (Masked Language Model) can be seen as Transformer’s generalization of “bidirectional modeling” - ensuring that bidirectional attention does not leak the target through mask.

Teacher forcing

There is a subtle choice when training a Seq2Seq / RNN Decoder - what is the input when decoding step $t$?

Problems with Teacher Forcing——Exposure Bias: The model has never “seen” its own wrong predictions during training. Once a certain step of prediction is biased during inference, it will go further and further wrong (the prototype of error accumulation/hallucination).

Modern solution:

• Scheduled Sampling: Gradually mixed into model prediction in the later stage of training
• Minimum Risk Training / REINFORCE: directly use the overall quality of the generated sequence as the training signal
• Transformer + Causal Masking: Compute all positions in parallel at once (see §4.3). It is still essentially Teacher Forcing, but the efficiency is much higher than RNN stepwise recursion.

Representative work: GRU (2014, a simplified version of LSTM), Seq2Seq (2014), Attention + LSTM (2015), Bidirectional LSTM (proposed by 2005, BiLSTM-CRF 2016 widely used)

4.3 Transformer

Transformer (Vaswani et al., 2017) has completely changed NLP and even the entire deep learning landscape. It completely abandons loops and convolutions, and only directly models the global dependence between any two positions in the sequence through the self-attention mechanism (Self-Attention).

Core: scaled dot-product attention

• Query (Q): represents “what am I looking for” (such as the current word).
• Key (K): stands for “what characteristics do I have” (used to match Q).
• Value (V): represents “the information I want to transmit” (the content after successful matching).
• $\sqrt{d_k}$ Scaling: When the dimension is large, the dot product result may be extremely large, causing softmax to enter a region with extremely small gradients. Scaling keeps gradients stable.

A specific example: the sentence “The cat sat on the mat”

Let’s say the model is processing the verb “sat” and it needs to find “who sat down” (the subject). The weights below are for visual illustration only and are not the output of a real model:

• The dot product of Qofsat and Kofcat is the largest (matching “singular noun subject”), and the attention weight after softmax is $\approx 0.7$
• There is also a certain match (noun) between Qofsat and Kofmat, but the semantic role is wrong and the weight is $\approx 0.15$
• The final output of sat $\approx 0.7 \cdot V_\text{cat} + 0.15 \cdot V_\text{mat} + \ldots$ - mainly absorbs the semantics of cat

This is how Attention works: each token broadcasts a “query”, all tokens respond with “How many do I match”, and finally the aggregated information is weighted by the degree of match. Q/K/V are generated from three different linear projections $W_Q, W_K, W_V$ from the same input, this flexibility allows the model to learn a variety of complex information routing patterns.

Multi-head attention

Map the input to $h$ different subspaces to calculate the attention separately, and then splice the results: \(\text{MultiHead}(Q,K,V) = \text{Concat}(head_1, \ldots, head_h) W^O\) Intuition: Different “heads” can focus on different information at the same time (for example, one head focuses on grammatical relationships and the other on semantic relationships).

Schematic diagram of the

Attention variants

As the context of large models becomes longer, the GPU memory usage of KV Cache (which caches K and V of all historical tokens during inference) becomes a bottleneck. For this reason, the multi-head structure of Attention has evolved into multiple variants, the core of which are reducing the number of K/V heads to compress the KV Cache:

flowchart LR
    subgraph MHA ["MHA: every Q Head independent KV"]
        Q1["Q₁"] --> KV1["K₁,V₁"]
        Q2["Q₂"] --> KV2["K₂,V₂"]
        Q3["Q₃"] --> KV3["K₃,V₃"]
        Q4["Q₄"] --> KV4["K₄,V₄"]
    end
    subgraph GQA ["GQA: Group sharing KV(g=2)"]
        Q1G["Q₁"] --> KVA["K,V group A"]
        Q2G["Q₂"] --> KVA
        Q3G["Q₃"] --> KVB["K,V group B"]
        Q4G["Q₄"] --> KVB
    end
    subgraph MQA ["MQA: all Q Share 1 group KV"]
        Q1M["Q₁"] --> KVS["K,V Share"]
        Q2M["Q₂"] --> KVS
        Q3M["Q₃"] --> KVS
        Q4M["Q₄"] --> KVS
    end

Why GQA is often used: MQA presses K/V to the 1 group, which may cause quality loss; MHA’s KV Cache is larger. GQA is a compromise between quality and GPU memory - e.g. LLaMA-2 70B uses 8 groups, 64 Q heads, and the KV Cache is about MHA’s 1/8; Actual gains still depend on model and task.

MLA’s innovation: DeepSeek uses a low-rank matrix to compress K/V into a $\sim$128-dimensional latent space. Only the latent vector is cached during inference, and the complete K/V is restored when needed. GPU memory has significant advantages in extremely long contexts.

Key components

1. Positional Encoding

Why does need position encoding? The core of Transformer - the self-attention mechanism is permutation equivariant (Permutation Equivariant) when there is no position encoding.: The order of input tokens changes, and the output will also be rearranged in the same way. When calculating $O = \sum \alpha_i V_i$, the model has no additional clues to determine the order, so the sentences “you hit me” and “I hit you” cannot be reliably distinguished, and position information must be injected.

Absolute position encoding: Sinusoidal design The original Transformer constructs a position vector using predefined sine and cosine functions: \(PE_{(pos, 2i)} = \sin\left(\frac{pos}{10000^{2i/d_{model}}}\right)\) \(PE_{(pos, 2i+1)} = \cos\left(\frac{pos}{10000^{2i/d_{model}}}\right)\)

Intuition: multi-frequency pointer (Clock Hands) We can think of each pair of $(\sin, \cos)$ as a “pointer” on a two-dimensional plane.

• low dimension ($i$ is small): high frequency, the pointer rotates extremely fast. Similar to the “second hand” of a clock, a few Tokens will complete a circle.
• high dimension ($i$ is large): low frequency, pointer rotation is extremely slow. Similar to the “hour hand”, it may take tens of thousands of Tokens to make one revolution. Transformer is like observing a dashboard composed of dozens of pointers with different rotation speeds, thereby accurately locking the absolute position of the current Token in the sequence.

Mathematical properties: Modeling relative position The subtlety of the author’s choice of sine and cosine functions is that it allows the model to be expressed through linear transformations relative position . According to the angle formula of trigonometric functions, there is an equation that is only related to the relative distance $r$ The relevant transformation matrix $M_r$ , making: \(PE_{pos+r} = M_r \cdot PE_{pos}\) This means that the model can more easily capture the distance information between two words instead of just the absolute coordinates when calculating Attention.

RoPE Position coding: Rotary Position Embedding design

Absolute position encoding adds position information to the Input Embedding, which indirectly affects Attention; ALiBi forcibly subtracts the distance deviation from the Attention Score. RoPE chose another path: directly encodes the position information into the vector itself in a rotational manner before doing the inner product of Q and K.

Core idea: Use rotation instead of addition

For each two dimensions of the vector ($d_0, d_1$), RoPE treats it as a vector on a two-dimensional plane, and then rotates the corresponding angle according to the location of the token $m$:

The rotation matrix is a standard two-dimensional rotation matrix:

For the $d$-dimensional vector, all dimensions are grouped into two groups, a total of $d/2$ groups, each group uses a different base angle $\theta_i$:

This design is in line with Sinusoidal PE - different dimensions correspond to different frequencies, low dimensions rotate quickly and high dimensions rotate slowly, which together form a unique position fingerprint.

RoPE How to naturally encode relative position?

When Q is at position $m$ and K is at position $n$, the inner product of the two is:

The inner product result only relies on the relative distance $m-n$ rather than the absolute coordinates. This is the neat thing about RoPE - there is no need to explicitly model relative distances like in Relative PE, the geometric nature of the rotation guarantees this automatically.

The relative position properties of

Geometric Proof of Translation Invariance

Assume that “cat” is at position 1 and “fish” is at position 3, calculate the Attention value $A$. Now insert 100 unrelated tokens in front, “cat” changes to position 101, and “fish” changes to position 103. The relative distance between the two remains unchanged (still 2), and the rotation angle $(m-n)\theta$ in the inner product remains unchanged, so the Attention value is still equal to $A$.

The geometry is more intuitive: Q rotates $N\theta$, K rotates $N\theta$, both rotate synchronously, and the inner product (the cosine of the included angle) remains unchanged.

Compatibility of with engineering acceleration

RoPE only modified Q and K itself. The calculation process of Attention is exactly the same as the original version, so it is naturally compatible with all Attention acceleration technologies:

• Flash Attention: directly available without modifying the operator;
• KV Cache: Directly cache the rotated $\text{K}^{(m)}$, read it out and use it without injecting the position again.

This is also one of the key engineering reasons why RoPE ultimately won - it is not only effective, but also perfectly compatible with the entire engineering ecosystem.

Common misunderstandings clarified

Many people think that RoPE guarantees “the farther the distance between Q and K, the smaller the Attention” like ALiBi. Actually RoPE does not guarantee this - rotation will produce a periodic oscillation pattern, and Attention fluctuates in a zigzag manner with distance.

This is an advantage: RoPE allows the model to learn “although distant, but still highly relevant” Attention patterns (such as long-range dependence), while ALiBi hard suppresses long-range attention.

Evolution of position encoding:

2. Point-wise feedforward network (Point-wise FFN) After each attention layer, a fully connected block (usually $d_{model} \to 4d_{model} \to d_{model}$) is followed to introduce nonlinear transformation: \(\text{FFN}(x) = \text{max}(0, xW_1 + b_1)W_2 + b_2\)

3. Residual connection and normalization (Add & Norm) Each sub-layer uses residual connections and cooperates with layer normalization (LayerNorm). There are currently two mainstream layouts:

• Post-LN (original version): First calculate the sub-layer and then add the residual to make LN. Good performance but difficult to train at deep level.
• Pre-LN (commonly used in modern large models): Do LN first and then calculate the sub-layer, which is usually easier to train stably; whether Warmup is needed still depends on the model, optimizer and training settings.

Encoder vs. decoder

Transformer uses an encoder-decoder architecture. The core difference between the two is the masking (Masking):

• Encoder: Bidirectional attention, each word can see all words in the sequence.
• Decoder: Use Masked Self-Attention to ensure that only the first $t-1$ words can be seen when generating the $t$ word (to prevent information leakage); it also contains Cross-Attention, used to pay attention to the output of the Encoder.

The specific form of causal mask (Causal Mask) - a lower triangular matrix of $n \times n$ (taking the sequence length 4 as an example):

implements: before softmax, set the attention score of the mask position to $-\infty$ (actually use a large negative number such as -1e9), and after softmax, the weight of these positions becomes 0. This ensures the causality of “the $t$ token can only see the position of $\leq t$”, and also realizes parallelization of Teacher Forcing during training - no longer recursive step by step like RNN, but forward prediction of the next token at all positions at once.

Attention visualization

One of the greatest charms of Attention is Interpretability ——The attention weight of each head in each layer can be directly visualized as a heat map, revealing the “attention mode” of the model. Typical findings:

Tools such as BertViz (Vig, 2019) and attention-flow (Abnar & Zuidema, 2020) can interactively display these patterns. Clark et al., 2019 “What Does BERT Look At?” systematically analyzed BERT’s 144 headers and found that a large number of headers capture clearly interpretable syntax/semantic functions - this is important evidence that Transformer transcends the “black box” label.

RNN vs Transformer comparison

Representative work: BERT (2018, pure Encoder), GPT series (2018-to date, pure Decoder), T5 (2019, Encoder-Decoder), ViT (2020, treat image blocks as sequences).

Vision Transformer (ViT)

ViT (Vision Transformer, Dosovitskiy et al., 2020) proved for the first time that pure Transformer can directly defeat CNN in processing image tasks with sufficient data, ending the nearly ten-year dominance of CNN in the field of vision.

Core Design: Treat images as “sentences”

flowchart LR
    IMG["Original picture<br/>224×224×3"] --> PATCH["Chunking<br/>16×16 patch"]
    PATCH --> FLAT["flattened to 196 a<br/>768 dimensional vector"]
    FLAT --> EMBED["Patch Embedding<br/>(linear projection)"]
    EMBED --> POS["+ Position Embedding<br/>+ [CLS] token"]
    POS --> TRANS["Transformer<br/>Encoder × L"]
    TRANS --> CLS["take [CLS] output"]
    CLS --> HEAD["MLP Head<br/>Classification"]
    style IMG fill:#e1f5ff
    style HEAD fill:#c4ffc4

Key steps:

1. Patch Split: Cut the $224 \times 224$ image into $14 \times 14 = 196$ patches of $16 \times 16$
2. Patch Embedding: Each patch is flattened into a $16 \times 16 \times 3 = 768$ dimensional vector, and the token is obtained through linear projection
3. [CLS] Token: Add a learnable “[CLS]” token before the sequence, and its final output is used for classification
4. Position Embedding: Add learnable position encoding (ViT uses learnable PE instead of Sinusoidal)
5. Transformer Encoder: Standard Transformer stack, usually 12–32 layers
6. classification head: take [CLS] output and connect to MLP

ViT inductive bias and data

Empirical conclusion (ViT original paper):

• Data < 1M (such as ImageNet-1k): ViT is significantly worse than ResNet
• Data ~14M (ImageNet-21k): ViT ties ResNet
• Data ~300M (JFT-300M): ViT significantly outperforms

This verifies a core principle of deep learning: less inductive bias × more data = more flexible architecture and higher upper limit.

The evolution of ViT

ViT not only replaces CNN as the visual backbone, but more importantly allows images and text to share the same Transformer architecture - this is the basis for large multi-modal models such as CLIP, LLaVA, and GPT-4V to unify vision and language in a concise way.

4.4 Graph Neural Network (GNN)

CNN processes grid data and RNN/Transformer processes sequences, but in reality a large amount of data is a graph (Graph) structure - a user-item bipartite graph of social networks, molecular structures, knowledge graphs, and recommendation systems. This type of data has a variable number of nodes, no fixed order, and an uncertain number of neighbors. The regular structure assumptions of CNN/RNN no longer hold true. Graph Neural Network (GNN) is specially designed for this type of non-Euclidean data.

Message passing

The unified paradigm of GNN is neighborhood aggregation: each node repeatedly “collects neighbor information → aggregates → updates its own representation”. The update of layer $l$ can be written as:

Here, \(\mathcal{N}(v)\) is the neighbor set of node $v$, $\text{AGG}$ is the aggregation function (sum/mean/maximum/attention weighting) of permutation-invariant, and $\sigma$ is nonlinear. After stacking $L$ layers, each node can perceive the subgraph structure within $L$ hops - this is the same as CNN expanding the receptive field layer by layer.

flowchart LR
    U1["neighbor u₁<br/>express h(u₁)"] --> AGG["aggregation AGG<br/>Sum/mean/attention"]
    U2["neighbor u₂<br/>express h(u₂)"] --> AGG
    U3["neighbor u₃<br/>express h(u₃)"] --> AGG
    V0["node v<br/>old representation h(v)"] --> UPD["update<br/>σ(W·agg + B·h(v))"]
    AGG --> UPD
    UPD --> V1["node v<br/>new representation h(v)′"]
    style V0 fill:#c4e1ff
    style V1 fill:#c4ffc4
    style AGG fill:#ffe1c4

Classic GNN comparison

Task types and core challenges

GNN Three types of tasks: node level (node classification, such as determining the topic of a paper in a citation network), edge level (link prediction, such as recommendation system), graph level (whole graph classification, such as molecular property prediction).

Main challenge - Over-smoothing: When the number of layers is too deep, all nodes express convergence and lose discriminability. Therefore, GNN usually only uses the 2–4 layer, which is much shallower than CNN/Transformer. Residual connections, jump knowledge (JKNet), PairNorm, etc. are common mitigation methods.

The relationship between and Transformer: Transformer can be regarded as GNN defined on a fully connected graph - each token is a node, and self-attention aggregates all other nodes with attention weight. GAT’s neighbor attention has the same origin as Transformer’s self-attention. In recent years, Graph Transformer (such as Graphormer) has migrated position coding and global attention to graphs, achieving SOTA in graph-level tasks such as molecules.

Representative work: GCN (2017), GraphSAGE (2017), GAT (2018), GIN (2019), Graphormer (2021); applications cover AlphaFold (protein residue contact graph), drug discovery, traffic prediction and recommendation systems.

5. Training optimization technology

5.1 Gradient Descent and Optimizer

The updated formula of the standard gradient descent is:

Here, $\eta$ is the learning rate, and $g_t$ is the gradient of Loss to the parameters. In this way, all parameters share the same learning rate. However, in the actual loss surface, the gradients in different directions are very different, and it is difficult to take into account the fixed learning rate. This gave rise to the adaptive optimizer.

Momentum

Momentum (Polyak 1964, re-developed by Sutskever et al., 2013 in deep learning) simulates the inertia of a small ball rolling down a hillside in physics:

in $\beta \in [0, 1)$ is the momentum coefficient (typical value $0.9$ ), $v_t$ is gradient exponential moving average ——Equivalent to “inertia”.

Physical intuition:

Nesterov Accelerated Gradient (NAG) is an improved version of momentum: it first “takes a step ahead” according to the current momentum and calculates the gradient at the predicted position, making the inertia more sensitive to the upcoming slope - “look-ahead momentum”.

Enabled in PyTorch via torch.optim.SGD(momentum=0.9, nesterov=True).

Optimizer evolution

flowchart TD
    SGD["SGD<br/>stochastic gradient descent<br/>1951"] -->|"Add momentum"| Momentum["Momentum SGD<br/>Suppress oscillation"]
    SGD -->|"Adaptive LR"| Adagrad["Adagrad 2011<br/>Cumulative gradient squared"]
    Adagrad -->|"exponential moving average"| RMSprop["RMSprop 2012<br/>Adapt to unstable states"]
    Momentum -->|"Combine"| Adam["Adam 2014<br/>First + second moments"]
    Adam -->|"Decouple weight decay"| AdamW["AdamW 2017<br/>Common baseline"]
    AdamW -->|"Curvature preconditioning"| Sophia["Sophia 2023<br/>Diagonal curvature estimate"]
    AdamW -->|"Fast and slow momentum"| AdEMAMix["AdEMAMix 2024"]
    Momentum -->|"Orthogonalize matrix updates"| Muon["Muon 2024"]
    Shampoo["Shampoo 2018<br/>Matrix preconditioning"] -->|"Adam in the eigenbasis"| SOAP["SOAP 2024"]
    Muon -->|"Update scaling and weight decay"| Moonlight["Moonlight 2025<br/>Large-scale training validation"]
    Muon -->|"Neuron-wise adaptive scaling"| NorMuon["NorMuon 2025<br/>ICML 2026"]

The arrows illustrate related ideas. Muon orthogonalizes momentum updates for matrices; it should not be understood as a switch added to AdamW. Methods based on norm geometry, such as Scion, are discussed below.

Classical and recent optimizers

A second moment is not a second derivative: Adam’s $v_t$ is a moving average of squared gradients, not a Hessian. Sophia estimates curvature; Shampoo / SOAP use matrix preconditioning; Muon changes the singular-value structure of matrix updates. These approaches address different aspects of optimization and cannot be adequately described by a single “adaptive LR” label.

Adam’s core formula (Kingma & Ba, 2014):

Standard hyperparameters: $\beta_1=0.9$, $\beta_2=0.999$, $\epsilon=10^{-8}$. Adam’s $m_t$ is responsible for the direction (can cross the saddle point); $v_t$ is responsible for the size (adaptive adjustment of the learning rate of each parameter). The two complement each other and jointly solve the two core problems of gradient descent.

Muon: from coordinate-wise scaling to matrix updates

Muon (MomentUm Orthogonalized by Newton-Schulz) treats hidden-layer weights as matrices and approximately orthogonalizes their momentum. Intuitively, AdamW adjusts each coordinate’s step size, while Muon rebalances different singular directions of a matrix update. Omitting Nesterov momentum, shape-dependent scaling, and weight decay, an illustrative formulation is:

The compact SVD expresses the ideal polar factor. Implementations approximate this transformation with a few Newton–Schulz iterations, avoiding an explicit SVD at every step. For a full-rank rectangular matrix, the ideal factor is semi-orthogonal; a finite approximation need not be strictly semi-orthogonal. The update direction is orthogonalized, rather than requiring the weight matrix itself to remain orthogonal. See the author’s explanation and implementation for the original algorithm and parameter grouping conventions.

Scaling to larger models: Moonlight adds weight decay and update-scale calibration to Muon and validates it on a MoE with 3B active / 16B total parameters, trained on 5.7T tokens. The paper reports roughly twice the compute efficiency in its comparisons. This does not imply that arbitrary end-to-end training runs take half the time, but it does establish evidence beyond small and medium-sized models. Original paper

Parameter grouping: Common implementations use Muon for hidden-layer matrices and AdamW for embeddings, output heads, biases, and normalization parameters. Grouping solely by p.ndim == 2 is insufficient: embeddings and output heads are also matrices, and tied weights must be deduplicated. Convolution-kernel flattening, whether Q/K/V are orthogonalized separately, and communication for sharded parameters should follow the conventions of the chosen implementation. Author’s implementation

SOAP: Adam in a better coordinate system

SOAP (Shampoo with Adam in the Preconditioner’s eigenbasis) accumulates left and right statistics of matrix gradients, periodically obtains their eigenbases, transforms gradients into that coordinate system, applies Adam there, and transforms updates back. It combines Shampoo’s matrix structure with Adam’s coordinate-wise adaptation. Original paper

It is inaccurate to state that “SOAP is equivalent to Shampoo.” The theoretical connection relies on particular assumptions, and practical behavior also depends on basis-update frequency, block size, and momentum handling. For an $m\times n$ matrix, the unblocked left and right statistics alone require $O(m^2+n^2)$ storage, with additional eigendecomposition costs. Fewer training steps must therefore be assessed alongside time per step and peak memory.

AdEMAMix, Scion, and NorMuon: three directions for improvement

• Retaining older gradient information—AdEMAMix (2024): A fast EMA tracks recent directions, while a slow EMA retains a longer history; a mixing coefficient combines them. The paper validates benefits in language modeling and image classification. This generally adds a per-parameter momentum buffer relative to AdamW, so it should not also be presented as saving optimizer-state memory. Original paper
• Changing the geometry of updates—Scion (2025): Norms are chosen according to layer input and output scales, and a linear minimization oracle produces updates. The framework includes constrained and unconstrained variants. One goal is better hyperparameter transfer across model sizes; this does not mean that every model requires no tuning. ICML 2025 paper
• Restoring neuron-wise adaptation—NorMuon (2025; ICML 2026): It introduces per-neuron second-moment statistics and row normalization after Muon orthogonalization to address uneven neuron updates, with particular attention to larger batches. It combines matrix updates with neuron-wise adaptation, rather than simply applying coordinate-wise Adam to Muon. ICML 2026 paper

Recent work in 2026: two directions to follow

As of this update, both are presented as recent preprints. AF-Muon’s reported savings of about 20% concern optimizer-state memory relative to its Hybrid Muon baseline, not a 20% reduction in total training memory. Beyond new optimizer names, useful questions include how vocabulary matrices are handled, how neuron-wise adaptation is retained after orthogonalization, and how orthogonalization and communication costs can be reduced.

Practical selection and fair comparisons

These are selection recommendations based on the mechanisms and available experimental evidence. Compare optimizers using the same data, model, token / sample budget, and validation protocol, with a reasonable tuning budget for each. Report GPU time to target quality, final quality, peak memory, and stability. Fewer steps, fewer tokens, fewer FLOPs, and less elapsed time are different metrics. Benefits in pretraining do not automatically transfer to LoRA, reinforcement learning post-training, or other modalities.

5.2 Learning rate scheduling

A fixed learning rate can be too large, causing oscillation, or too small, causing slow progress. Warmup + Cosine Decay is a common approach; WSD and Schedule-Free reduce dependence on a predetermined training endpoint in different ways.

Other commonly used schedulers

WSD: separating the training trunk from final convergence

WSD (Warmup–Stable–Decay) maintains a constant learning rate after warmup. Once an endpoint for a particular budget is chosen, training branches from a trunk checkpoint into a decay phase. Unlike a cosine schedule tied to a total step count from the start, this allows the stable-phase trunk to be reused to produce models at different compute budgets. WSD analysis paper

This is still a learning rate schedule: the peak rate, warmup, decay duration, and decay shape must be chosen, and decay consumes compute. A stable-phase checkpoint and one that has completed cooldown should not be treated as having equal budgets and convergence states.

Schedule-Free: iterate averaging instead of a preset decay curve

Schedule-Free (2024) combines momentum with iterate averaging, distinguishes the gradient-evaluation point from the parameters used for evaluation, and avoids a decay curve tied to a predetermined stopping step. Variants include SGD and AdamW. Its mechanism differs from WSD, and learning rate, warmup, and weight decay still require configuration. Original paper

Implementation details: The authors’ non-closure interface requires optimizer.train() during training and optimizer.eval() before validation or saving evaluation weights, alongside model.train() / model.eval(); switching model mode alone is insufficient. Resumable checkpoints should retain optimizer state according to the implementation’s instructions. BatchNorm also requires handling running statistics for the evaluation parameters. The official implementation still recommends warmup. Author’s repository

selection principle:

• trains from scratch CNN: StepLR or Cosine Annealing (Warm Restart for long training)
• fine-tuned pretrained model: Warmup + Linear/Cosine (low peak LR, usually $1-5 \times 10^{-5}$)
• LLM pretraining: Compare Warmup + Cosine for fixed budgets and WSD when the budget may grow. Calibrate peak learning rate and warmup to model scale, batch size, and optimizer rather than copying them across configurations
• validation set available and uncertain number of epochs: ReduceLROnPlateau
• Rapid Prototyping/Competition: 1Cycle (Smith said it can achieve “super-convergence” effect)

5.3 Parameter initialization

Different initial parameters $\theta_0$ may lead to convergence to different local optimal solutions.

Xavier initializes (Glorot & Bengio, 2010, suitable for symmetric activation such as Sigmoid/Tanh):

This method takes into account the variance stability of forward activation and reverse gradient, and is the default choice for symmetric activation functions.

Kaiming initializes (He et al., 2015, suitable for ReLU activation):

ReLU will set half of the input to zero, and Xavier’s variance assumption no longer holds, so Kaiming amplifies the variance 2 times to compensate. This initialization keeps the activation value variance stable in the early stages of dozens of layers of ReLU network training, preventing it from exploding or disappearing.

Selection Principle: Sigmoid/Tanh commonly used Xavier, ReLU and its variants commonly used Kaiming; the actual initialization also needs to be combined with the residual structure, normalization and framework default implementation.

pre-training as initialization (Pre-training): train on large-scale data first, and then transfer parameters to the target task. Improve Optimization (better starting point) and Generalization (learn common features) at the same time.

5.4 Normalization method

Normalization (Normalization) forces the output of each layer of the network to be within a reasonable range, making the loss surface flatter and the learning rate easier to adjust.

Comparison of normalization methods

Batch Normalization(BN, Ioffe & Szegedy, 2015):

Here, $\gamma, \beta$ is a learnable scaling and translation parameter, which gives the model the ability to “cancel normalization”.

Key differences between BN training and inference

BN behaves differently during training and inference, which is the most common pitfall in engineering practice:

Common bug: In PyTorch, model.eval() will automatically enable running stats when switching to inference mode; if you forget to call it, the current batch statistics will still be used during inference, resulting in randomization of the output when batch size=1.

Limitations of BN and the necessity of LN

BN has three fatal shortcomings, which correspond to the three reasons why Transformer chose LN:

Layer Normalization(LN, Ba et al., 2016):

Layer normalization normalizes across the feature dimensions of each sample, independently of batch size and sequence length. This is why it is standard in Transformers.

normalized dimension compared to intuition (taking the 4D feature map with shape [N, C, H, W] as an example):

RMSNorm (2019): Remove mean normalization and only retain variance scaling:

Typically slightly less computationally expensive than LN, and similar in many language models; adoption depends on architecture and training recipe.

5.5 Residual connection

Deep networks (100+ layers) face the coexistence problem of gradient vanishing and exploding gradients. Skip Connection/Residual Connection (He et al., ResNet, 2015):

That is, the input $x$ (identity mapping) is directly superimposed on the original transformation $F(x)$. Even if the effect of $F(x)$ is weak, the gradient can still flow directly back to the shallow layer through the constant path, which greatly alleviates the vanishing gradients and enables stable training of extremely deep networks such as ResNet-152.

Variants of residual connections

After ResNet, the residual connection developed multiple variants, each weighing “identity degree” and “information density”:

Key insights of Pre-activation ResNet (v2): ReLU in ReLU(F(x)+x) of the original ResNet will truncate part of the gradient; v2 moves BN-ReLU inside $F$, the residual path is completely identity mapping, and the gradient penetrates losslessly. This enables stable training of ResNet-1001 (1001 layer!). Transformer’s Pre-LN layout is essentially a generalization of the same idea.

DenseNet offers a counterintuitive result: each layer concatenates all preceding layers, which seems likely to cause an explosion in parameter count. Feature reuse allows each layer to remain narrow (often with a growth rate of only 32 channels), so the total parameter count can be smaller than that of a ResNet of the same depth. The trade-off is high GPU memory use because many intermediate activations must be retained.

Skip Connection has become a standard component of almost all architectures in modern deep learning (ResNet, Transformer, U-Net, Diffusion UNet). What is improved is Optimization - it does not solve “whether deep networks can express complex functions” (shallow layers can also do so), but “whether deep networks can be trained by SGD”.

6. Loss function and regularization

6.1 Loss function

Loss function (Loss Function) is the first step of the three-step framework of machine learning, which defines “how to measure the quality of the model.” Choosing an appropriate loss function is crucial for model training.

Mean Squared Error (MSE)

Mean Squared Error (MSE) is suitable for regression task, which measures the mean difference between the predicted value and the true value:

• Advantages of **: differentiable everywhere, simple gradient calculation; heavier penalty for larger errors (square amplification effect)
• Disadvantages: Extremely sensitive to outliers; gradients may explode when the prediction error is large

Mean Absolute Error (MAE)

• Advantages: More robust to outliers (linear penalty instead of square)
• Disadvantages: Not differentiable at \(y_i = \hat{y}_i\) (need to use Huber Loss to compromise)

Huber Loss

Use MSE (smooth and differentiable) when the error is small, use MAE (anti-outlier) when the error is large, and $\delta$ controls the switching threshold. Regression branch commonly used for object detection.

Cross-Entropy loss (Cross-Entropy)

Cross-Entropy Loss (Cross-Entropy Loss) is suitable for classification tasks, including image classification and language model (the nature of the next token prediction is multi-classification).

The first step: Convert the logits output by the network into probability distribution through Softmax:

The second step: Calculate the cross entropy with the real label (take the logarithmic probability of the real category, add a negative sign):

Here, $y^*$ is the real category, and \(\hat{p}_i\) is the one-hot label.

Why not use accuracy as Loss? The accuracy is a step function. When the parameters change slightly, the Loss is almost always zero, the gradient cannot be calculated, and Gradient Descent cannot be performed. Cross-Entropy is differentiable everywhere, and the smaller the value, the higher the Accuracy.

numerical stability: In actual implementation, Softmax and Cross-Entropy are combined (LogSoftmax + NLLLoss) to avoid $e^{z_i}$ numerical overflow.

Label Smoothing

Label Smoothing (Szegedy et al., 2016, widely popularized by Inception-v3 with the original Transformer) is a simple but powerful regularization trick for Cross-Entropy.

motivation: one-hot label requires the model to predict the correct category 1.0, the others are 0 - this forces the model to push the logit to infinity to fully fit, resulting in overconfidence (overconfidence), generalization Poor, distillation effect is weak.

approach: soften the hard label \(\hat{p}_i\) to:

Here, $K$ is the number of categories, and $\epsilon$ is the smoothing coefficient (typical value $0.1$). The expected probability of the correct class drops to $0.9$, and the remaining $0.1$ is evenly distributed to other classes.

effect:

• Suppress model overconfidence, calibration (calibration) is better - that is, the actual accuracy when predicting 90% is also close to 90%
• Slightly improved generalization (approximately 0.2–0.5% higher Top-1 accuracy on ImageNet)
• Improve teacher soft label quality for knowledge distillation (soft distribution carries more information than one-hot)

Note: Label Smoothing will lose a small amount of confidence information, which needs to be weighed in tasks that require confidence selection (active learning, rejection).

Binary cross-entropy (BCE)

Two classification tasks (using Sigmoid for the output layer):

Focal Loss

Focal Loss (Lin et al., 2017, proposed in RetinaNet) is specially designed for scenes with serious imbalance in the category - for example, there are far more background boxes than foreground boxes in target detection (1: 1000), and there are few positive samples in medical images.

Standard Cross-Entropy still gives a non-negligible loss for easy classification samples (prediction probability $p \to 1$), and a large number of easy samples will “swamp” the gradient of a small number of difficult samples. Focal Loss is multiplied by a modulation factor $(1-p)^\gamma$ in front of CE:

• $\gamma$ (focusing parameter, typical value 2): When $p \to 1$ $(1-p)^\gamma \to 0$, almost no gradient is generated; when $p \to 0$ (difficult sample) the factor approaches 1, gradient is normal
• $\alpha$ (class balancing, typical value 0.25): Static adjustment of positive and negative class weights

Effect: For the first time, RetinaNet allowed a single-stage detector to equal the accuracy of the two-stage Faster R-CNN, and Focal Loss played an important role. Since then, it has been widely used in long-tail classification, anomaly detection, and medical image segmentation.

KL divergence

KL Divergence (Kullback-Leibler Divergence) measures the difference between two probability distributions, often used in knowledge distillation and variational autoencoders (VAE):

Mathematical relationship between cross entropy and KL divergence

Cross-Entropy and KL divergence are not two independent losses; Different perspectives of the same quantity :

where $H(P) = -\sum_i P(i) \log P(i)$ is the entropy of the true distribution $P$.

infers: When $P$ is a one-hot hard tag, $H(P) = 0$, so CE $\equiv$ KL. So in standard classification tasks, “minimizing Cross-Entropy” and “minimizing KL divergence” are exactly equivalent.

Practical differences:

• Hard label (one-hot): using CE, simple calculation
• Soft labels (distillation, Label Smoothing, RLHF reference strategy): Use KL, because when $H(P) \neq 0$, CE contains a non-optimizable constant term, KL more cleanly measures the “distance from the target distribution”

Contrastive loss

Metric Learning (Metric Learning) The goal is to learn an embedding space so that “similar samples” are close and “heterogeneous samples” are far away. Two types of classic losses:

Contrastive Loss (Hadsell et al., 2006): Given a pair of samples $(x_i, x_j)$, the labels $y=1$ represent the same type and $y=0$ the heterogeneous one:

Here, $d$ is the embedding distance (usually Euclidean), and $m$ is the margin hyperparameter.

Triplet Loss (Schroff et al., 2015, FaceNet): Given a triplet (anchor $a$, positive $p$, negative $n$):

The intuition is that “the distance + margin between the positive sample and the anchor point is smaller than the distance between the negative sample and the anchor point”. FaceNet achieves 99.63% accuracy on LFW with its learned face embeddings.

InfoNCE (Contrastive Predictive Coding, Oord et al., 2018): The mainstream form of modern contrastive learning, one positive sample performs softmax on $N-1$ negative samples:

where $\tau$ is the temperature coefficient. SimCLR, MoCo, CLIP are all based on InfoNCE - this type of loss drove the explosion of self-supervised representation learning in 2020–2023.

InfoNCE is essentially Cross-Entropy of $N$ classification - positive samples are the only “correct category”, and negative samples are “other categories”. This once again confirms the universality of CE.

Loss function selection quick check

6.2 Dropout

Dropout (Srivastava et al., 2014): Set $p$ as the dropout rate, set the neuron output to zero with probability $p$ during training, and divide the retained activation by $1-p$ (inverted dropout). Turn off Dropout during inference and use the full network directly, so no additional scaling is required.

Intuition: forcing the network to predict correctly even in the absence of some neurons, preventing excessive co-adaptation between neurons, is equivalent to training a large number of sub-networks with different structures at the same time and obtaining the integration effect.

Timing of use: When there is a clear gap between training and verification errors, Dropout can be used as a regularization option; if the training Loss itself cannot be reduced, the optimization settings should be checked first. Too strong Dropout may further slow down convergence.

6.3 Data augmentation

Data Augmentation Artificially expands the amount of data by applying semantic-preserving transformations to training samples:

Mixup Formula: $\tilde{x} = \lambda x_i + (1-\lambda)x_j$, $\tilde{y} = \lambda y_i + (1-\lambda)y_j$

Note: Data augmentation transformations must maintain label semantics. If the task is to determine the direction of the bird’s head, left and right flipping is not possible; if the task is speaker identification, speaker switching is not possible.

Timing of use: Data augmentation is usually used to improve generalization and distribution robustness, but the enhancement intensity must match the task. If the Training Loss cannot be reduced, you should first confirm that the enhancement does not destroy the label semantics, and then check the optimization settings.

6.4 L2 regularization and AdamW

L2 regularization (Weight Decay) adds the L2 norm penalty of the parameters in Loss to make the optimization bias toward solutions with smaller absolute values of parameters (more “simple” functions):

AdamW (Loshchilov & Hutter, 2017) fixed a common error in adding L2 regularization to Adam: the traditional method merges the regularization gradient with the ordinary gradient and then scales them uniformly by Adam, causing the regularization effect to be “diluted” by the adaptive learning rate. AdamW instead directly performs Weight Decay on the parameters, and then performs Adam update:

θ = θ × (1 - lr × λ)       # Weight Decay Acts directly on parameters (without adaptive term scaling)
θ = θ - lr × Adam_update    # Adam Normal update

AdamW is the current standard optimizer for large language model training, and is usually used with gradient clipping.

6.5 Semi- and self-supervised learning

In real scenarios, labeled data is scarce and unlabeled data is cheap. Semi-supervised learning (Semi-supervised Learning) and self-supervised learning (Self-supervised Learning) are the two mainstream paths for utilizing unlabeled data.

Core idea

Relationship to modern large models

The pre-training of large language models is essentially the largest self-supervised learning - “predicting the next token” (Causal LM) or “predicting the masked token” (Masked LM) on massive unlabeled texts, and then adapting to downstream tasks through fine-tuning (SFT, RLHF) with a small amount of labeled data. The success of the Pre-training + Fine-tuning paradigm makes “scarcity of annotated data” no longer the main bottleneck for the implementation of deep learning.

7. Method classification summary

flowchart TD
    A["Neural network training techniques"] --> B["Step 3: Improve Optimization"]
    A --> C["Step 2: Improve network architecture"]
    A --> D["Step 1: Improve Loss with data"]
    B --> B1["AdamW / Muon / SOAP<br/>Coordinate-wise or matrix updates"]
    B --> B2["Cosine / WSD<br/>or Schedule-Free iterate averaging"]
    B --> B3["Kaiming Init<br/>Parameter initialization"]
    B --> B4["Pre-training<br/>improve at the same time Opt+Gen"]
    C --> C1["CNN<br/>Receptive Field + Parameter sharing<br/>improve Generalization"]
    C --> C2["Skip Connection<br/>Mitigating vanishing exploding gradientss<br/>improve Optimization"]
    C --> C3["BN / LN / RMSNorm<br/>Major improvements Optimization"]
    D --> D1["Cross-Entropy<br/>make Optimization feasible"]
    D --> D2["Dropout<br/>improve Generalization"]
    D --> D3["Data Augmentation<br/>improve Generalization"]
    D --> D4["AdamW / Weight Decay<br/>improve Generalization"]

Comparison table of objectives of each method:

7.1 Training troubleshooting quick check

When encountering a training abnormality, first determine the problem based on the curve, and then choose to make changes. By changing only a few factors at a time and recording the data, random seeds, and evaluation protocols, the results can be easily compared.

8. Commonly used experimental benchmarks

Benchmarks are tools for measuring progress. This section sorts out classic benchmarks by field - MNIST/ImageNet for CV, GLUE/MMLU for NLP, as well as tasks such as coding, mathematical reasoning, and long context. The models, data, prompts and evaluation scripts of different papers may be different. The scale and historical results in the table are suitable for establishing concepts and should not be directly used as cross-paper rankings.

8.1 Computer Vision Benchmark

MNIST

CIFAR-10 / CIFAR-100

The effects of Dropout, Batch Normalization, ResNet, and Data Augmentation have been extensively verified on this benchmark.

ImageNet(ILSVRC)

ImageNet Top-1 Accuracy evolution (see table in Section 4): AlexNet (56.5%) → VGG (71.5%) → ResNet (76.0%) → EfficientNet (84.4%). These numbers come from classic models and training recipes from different eras, and are mainly used to illustrate the evolution of the architecture.

8.2 NLP benchmarks

GLUE / SuperGLUE

BERT (2018) achieved leading results on GLUE when it was released; as the benchmark gradually became saturated, the researchers subsequently launched the more difficult SuperGLUE.

MMLU

MMLU is suitable for observing the knowledge coverage and multi-disciplinary reasoning of the model, but the scores will be affected by prompts, calibration methods and training data contamination, and the evaluation protocol should be fixed when comparing.

Language model perplexity (Perplexity)

Penn Treebank (PTB)/WikiText is the standard benchmark for traditional language models. The evaluation index is perplexity (PPL) - the lower the better, indicating that the model is more certain in predicting the next token. PPL is still suitable for comparing language models under the same corpus, word segmentation and modeling goals, but it cannot represent knowledge, code or reasoning ability alone.

Code Competency Benchmark

To evaluate LLM coding ability, it is required for almost all cutting-edge model releases.

Usage Suggestions: HumanEval/MBPP is suitable for testing function-level code generation, and SWE-bench is closer to real warehouse repair; when reporting results, the number of sampling times, test set version and whether tool calls are allowed should also be stated.

Mathematical Reasoning Benchmark

GSM8K is more focused on basic multi-step arithmetic, while MATH and AIME have higher requirements for competition mathematical reasoning. As model capabilities improve, training data contamination, sampling strategies, and answer verification methods will significantly affect performance, and you cannot just look at a single score.

Long context benchmark

As the context window expands from 2K→32K→128K→1M, simple perplexity can no longer reflect the long context capability of the model, and special benchmarks have emerged:

Usage Suggestions: Needle-in-a-Haystack mainly tests positioning capabilities and cannot replace real long document Q&A. Actual evaluation should also focus on location, document length, question type, retrieval interference, and accuracy of answer citations.

9. Efficient training and inference

9.1 Mixed precision training

Mixed-precision training uses formats such as FP16 / BF16 for suitable operations, while retaining higher precision for sensitive computations, accumulation, or master weights. Memory savings and speedups depend on hardware, operator coverage, activations, and optimizer states. Halving a tensor’s byte count does not imply halving total memory use.

BF16 (Brain Float 16) has a wider exponent field (8 bits versus 5 in FP16), giving it a larger numerical range. It has become the preferred low-precision format for large language model training.

FP8: training precision as part of system design

FP8 training depends on scaling, accumulation precision, and operator selection. The DeepSeek-V3 report uses fine-grained scaling and FP8 inputs for most Linear GEMMs, while retaining higher precision for normalization, routing, embeddings, output heads, and other sensitive components, and increasing precision for parts of the accumulation. Technical report

Long training runs can reveal problems that short experiments miss. Scaling FP8 Training to Trillion-Token LLMs analyzes SwiGLU outlier amplification over trillion-token runs and introduces Smooth-SwiGLU. FP8 should therefore be evaluated using loss, gradient norms, and downstream quality over the full training run, in addition to throughput. Simply casting BF16 tensors to FP8 does not establish equivalent results. Original paper

9.2 Gradient clipping

Occasional exploding gradientss (loss spikes) during large model training can lead to training crashes. Gradient clipping prevents this problem by limiting the upper bound on the L2 norm of the gradient:

The usual threshold is $\tau = 1.0$, which is the standard companion to AdamW.

9.3 Gradient checkpointing

During training, backpropagation requires all intermediate activation values in the forward process to calculate gradients - under deep networks, the activation value GPU memory can occupy several times the parameters themselves, becoming the main overhead of GPU memory for large model training.

Gradient checkpointing (Chen et al., 2016) retains activations only at selected checkpoint layers. During backpropagation, it recomputes the missing intermediate activations with another forward pass, trading approximately 33% additional computation for activation memory on the order of $O(\sqrt{N})$.

The use of gradient checkpoints in combination with mixed precision, ZeRO, and tensor parallelism is a standard GPU memory optimization method for training large models with hundreds of billions of parameters (gradient_checkpointing=True for HuggingFace Transformers, torch.utils.checkpoint for PyTorch).

9.4 Flash Attention

The bottleneck of standard attention is that the $N \times N$ attention matrix needs to be written to GPU HBM GPU memory. The GPU memory complexity is $O(N^2)$, which is extremely expensive for long sequences.

FlashAttention equivalently rearranges standard dense softmax attention, while still introducing finite-precision numerical error. It reduces IO and intermediate memory use, but does not change dense attention’s computational complexity from $O(N^2)$ to $O(N)$.

FlashAttention-4 results need their comparison conditions: On B200 with BF16 and the tested shapes, the paper reports up to roughly 1.3× speedup over cuDNN 9.13 and 2.7× over Triton. These are kernel results for specific versions and hardware, not end-to-end model-training speedups. Results from different generations should not be combined across hardware into a single speedup ranking. Original paper

10. Model compression and adaptation

10.1 Mixture-of-Experts (MoE)

Mixture of Experts(MoE) The core idea: split a large FFN into $N$ independent “expert”, led by an Router/Gating Network Select top- for each token $k$ Experts perform calculations. Only activating some experts means - The number of parameters is huge, but only a small part of them is used each time. .

flowchart LR
    X["Token<br/>hidden state"] --> R["Router<br/>(small MLP)"]
    R -->|"top-2 scores"| S["softmax Weighted"]
    X --> E1["Expert 1<br/>FFN"]
    X --> E2["Expert 2<br/>FFN"]
    X --> EN["Expert N<br/>FFN"]
    E1 -.->|"selected"| S
    E2 -.->|"selected"| S
    S --> Y["weighted output"]
    style E1 fill:#c4e1ff
    style E2 fill:#c4e1ff
    style EN fill:#f0f0f0

Core indicators

Representative work

Key challenge: Load balancing prevents tokens from concentrating on a few experts, often through auxiliary losses or capacity constraints. Total and active parameter counts are not reported uniformly across models; routing strategies and FFN sizes also matter. See the technical report for DeepSeek-V3’s configuration above and its load-balancing design.

10.2 Parameter-efficient fine-tuning

As the number of pretrained model parameters expands to tens or hundreds of billions, Full Fine-tuning GPU memory and storage costs tend to be high. Parameter-Efficient Fine-Tuning(PEFT) Only train a very small number of additional parameters (usually < 1% ), which can approximate the full parameter fine-tuning effect.

Comparison of mainstream PEFT methods

LoRA core formula

For pretrained weight $W_0 \in \mathbb{R}^{d \times k}$, LoRA freezes $W_0$ and only trains low-rank decomposition $B \in \mathbb{R}^{d \times r}, A \in \mathbb{R}^{r \times k}$ ($r \ll \min(d,k)$, typical $r=8, 16, 64$):

Forward calculation: $h = W_0 x + BAx$.

Advantages:

• trains: only $B, A$ requires gradients and optimizer state, GPU memory requirements drop to $\sim 1/100$
• storage: One task only needs to save $BA$ (MB level), a set of base molds can load LoRA for hundreds of tasks
• inference: $BA$ can be merged into $W_0$, with zero inference overhead
• switching task: load different LoRA weights to switch scenes in seconds

LoRA/QLoRA are common strong baselines for open source large model fine-tuning, but the best solution still depends on the task, base model, quantization method and available GPU memory.

10.3 Quantization

quantization compresses the model weights (and activations) from FP16/BF16 to INT8/INT4 or even lower bit width, with a small loss of accuracy in exchange for model size reduced to 1/4~1/8, inference speed improvement 2-4×.

Quantization types

Modern quantization methods

Experience: INT4 can often significantly reduce GPU memory usage, but the quality loss depends on the model, calibration data, quantization granularity and inference kernel; latency, throughput and accuracy should be measured on the target task before deployment.

10.4 Knowledge distillation

Knowledge Distillation (Hinton et al., 2015) migrates the capabilities of a large “teacher” model to a small “student” model, allowing the small model to achieve performance close to that of the large model during deployment.

Core idea: soft labels

The one-hot hard label only tells students “This picture is a cat”; while the soft probability distribution output by the teacher model can tell students: “This picture 90% is a cat, 8% is a dog, 1% is a fox, 0% is a car”—the relative relationship between these categories (“A dog is closer to a cat than a fox”) implies the teacher’s knowledge.

Distillation loss:

Here:

• $p_t^{(T)} = \text{softmax}(z_t / T)$ is the output softened by temperature $T$ for teachers (the distribution is smoother at $T > 1$)
• $p_s^{(T)}$ is the softened output corresponding to the student
• $\alpha$ Balancing hard label supervision and soft label distillation

Three Paradigms of Distillation

Representative work

In LLM, distillation can act on logits, hidden representations, preference data, or generate trajectories. It is often used in combination with quantization, pruning, or parameter sharing, but the capability gap between the teacher model and the student model, data distribution, and generation temperature all affect the final effect.

11. Frontier architectures

11.1 RLHF

RLHF (Reinforcement Learning from Human Feedback) is the key to the success of ChatGPT - aligning the pretrained large model from “speaking” to “understanding instructions and in line with human preferences”.

flowchart LR
    PT["pre-training LLM<br/>(GPT-3)"] --> SFT["Step 1: SFT<br/>Supervised fine-tuning"]
    SFT --> RM["Step 2: Training reward model<br/>Humans sort multiple answers"]
    RM --> PPO["Step 3: PPO reinforcement learning<br/>Maximize rewards"]
    PPO --> ALIGNED["Aligned model<br/>(ChatGPT)"]
    style PT fill:#e1f5ff
    style ALIGNED fill:#c4ffc4

Three-stage process

Alternatives to RLHF

DPO has become a common preference optimization method because the training process is simple and there is no need to separately train the reward model. PPO, DPO, rejection sampling, and other methods can also be used in combination; the specific training process of closed-source models is often not fully disclosed.

From preference alignment to reasoning training: GRPO and RLVR

GRPO (Group Relative Policy Optimization) was introduced in DeepSeekMath (2024). It samples a group of responses to the same question and uses within-group reward statistics to construct relative advantages, retaining a PPO-style clipped objective while removing the separately trained value / critic network. For the case of one outcome reward per response, the advantage can be illustrated as:

Here, $G$ is the number of samples per question, $r_i$ is the reward for response $i$, and $\epsilon$ provides numerical stability. Removing the critic does not remove sampling costs: generating multiple responses per question can dominate compute, and groups with identical rewards provide little relative discrimination. DeepSeekMath

RLVR (Reinforcement Learning with Verifiable Rewards) uses verifiable outcomes as rewards, such as checked mathematical answers or code tests. It describes the reward source, while GRPO / PPO describe policy-training algorithms; they can be combined. DeepSeek-R1 (2025) trains reasoning with rule-based rewards and combines cold-start data, reinforcement learning, and other training stages. R1-Zero and the full R1 pipeline should be distinguished. Technical report

This also completes the three-step framework: RLVR changes the training signal; GRPO / DAPO define policy objectives and the sampling and update procedure; underlying parameter updates still require optimizers such as AdamW. These methods operate at a different level from the gradient optimizers in §5. Gains on verifiable-task rewards should also be evaluated on independent test sets and other tasks to assess generalization.

11.2 Generative model: from AE to GAN

The discriminative model learns \(p(y\mid x)\) (given input prediction label), and the generative model learns the data distribution \(p(x)\) itself, so it can sample new samples “out of nothing”. There are four major families of deep generative models, which continue to evolve along the trade-off of “sampling quality/training stability/latent space controllability”: autoencoder (AE) → variational autoencoder (VAE) → generative adversarial network (GAN) → diffusion model (detailed in the next section).

Autoencoder (AE)

The encoder $f$ compresses the input into a low-dimensional latent vector $z$, the decoder $g$ reconstructs $\hat{x}$, and the training target is the reconstruction error \(\mathcal{L} = \|x - g(f(x))\|^2\). AE is good at dimensionality reduction / denoising / representation learning, but the latent space has no probability constraints and irregular distribution. cannot randomly sample from it to generate new samples - it is essentially a compressor, not a real generative model.

Variational autoencoder (VAE)

VAE adds probabilistic structure to the latent space: the encoder outputs a distribution \(q_\phi(z\mid x)\) (instead of a single point) and constrains it to be close to the prior \(p(z)=\mathcal{N}(0,I)\). Training to maximize evidence lower bound (ELBO):

The former term is the reconstruction term, and the latter term is the regular term that pulls the latent distribution toward the prior. Through the reparameterization trick \(z = \mu + \sigma \odot \epsilon\) (\(\epsilon \sim \mathcal{N}(0,I)\)), the sampling can be differentiated and the gradient can be returned. After training, it can be generated by directly sampling from the prior \(p(z)\). VAE sampling is stable, the latent space is continuous and interpolable, but the generated image is blurry.

Generative adversarial network (GAN)

GAN uses to counter the game to bypass the explicit likelihood: the generator $G$ maps noise into false samples, and the discriminator $D$ learns to distinguish between true and false, and the two alternately optimize a minimax objective:

Image generated by GAN Sharp and high quality (StyleGAN’s faces are almost lifelike), but training is known for being unstable—— mode collapse , the game is difficult to converge and requires a lot of techniques (WGAN, spectral normalization, etc.) to stabilize it.

List of four major families

flowchart LR
    AE["AE<br/>Compressed reconstruction"] --> VAE["VAE 2013<br/>probability latent space"]
    VAE --> GAN["GAN 2014<br/>confrontational game"]
    GAN --> DIFF["Diffusion 2020<br/>Denoising generation"]
    style AE fill:#e1f5ff
    style DIFF fill:#c4ffc4

The diffusion model combines the training stability of VAE and the high quality of GAN, and has become the mainstream of current generative modeling - the next section will expand.

11.3 Diffusion models

Diffusion model is the most successful generative paradigm in deep learning in recent years, driving epoch-making work such as Stable Diffusion, DALL-E 3, Midjourney, and Sora. It is another pillar of deep generative modeling alongside Transformer.

Core idea: noise addition and denoising

flowchart LR
    X0["x₀<br/>clear image"] -->|"Forward: progressively adding noise"| XT["xₜ<br/>pure noise"]
    XT -->|"Reverse: learning to denoise"| XM["x̂₀<br/>generate image"]
    style X0 fill:#c4ffc4
    style XT fill:#ffcccc
    style XM fill:#c4e1ff

forward process (Forward / Diffusion): gradually add Gaussian noise to the real data, and after $T$ steps (usually 1000) it becomes an approximate standard normal distribution:

Since the properties are deducible, $x_t$ can be obtained from $x_0$ in one step: \(x_t = \sqrt{\bar{\alpha}_t} \cdot x_0 + \sqrt{1-\bar{\alpha}_t} \cdot \epsilon\), where \(\bar{\alpha}_t = \prod_{s=1}^t (1-\beta_s)\).

Reverse process (Reverse / Denoising): Train a neural network \(\epsilon_\theta(x_t, t)\) to predict the noise added at each step. The training loss is:

reasoning , sampled from the standard normal distribution $x_T$ , then use \(\epsilon_\theta\) Gradually remove noise, $T$ After the step, the generated sample is obtained.

Why diffusion beat GANs

Representative work

From UNet to DiT

The early diffusion model uses UNet (encoder-decoder CNN with skip connection) as the denoising network. 2022 appeared DiT (Diffusion Transformer, Peebles & Xie, 2022) years later, using Transformer to replace UNet - Sora, Stable Diffusion 3 all use DiT. This once again confirms the trend of Transformer unifying multiple fields.

11.4 State space model and Mamba

Transformer’s attention mechanism affects sequence length $N$ The computational and GPU memory complexity of $O(N^2)$ , which is extremely expensive when processing long sequences. State Space Model (SSM) Provides an alternative architectural route with linear complexity.

SSM basics

SSM originates from cybernetics and maps the input sequence $x(t)$ to the output sequence $y(t)$ through the hidden state $h(t)$:

\(h'(t) = Ah(t) + Bx(t)\) \(y(t) = Ch(t)\)

Here, $A \in \mathbb{R}^{N \times N}$ is the state transition matrix, and $B, C$ is the projection matrix. After discretization (step size $\Delta$, zero-order preservation), the recursive form is obtained:

Among them $\bar{A} = e^{\Delta A}$, $\bar{B} = (\Delta A)^{-1}(e^{\Delta A} - I) \cdot \Delta B$.

SSM combines the advantages of RNN and CNN:

S4: structured state spaces

S4 (Gu et al., 2021) is the first practical deep SSM: restricting the matrix $A$ to HiPPO initialized diagonal plus low rank (DPLR) The structure makes long sequence training efficient and feasible, significantly surpassing Transformer on long sequence benchmarks such as Long Range Arena.

Limitations of S4: The $A, B, C$ matrix is fixed for all time steps (time-invariant) and cannot dynamically choose which information to focus on or ignore based on the input content - essentially a static filter.

Mamba (2023)

The core innovation of Mamba (Gu & Dao, 2023) is selectivity (Selectivity): convert $B, C$ and step size $\Delta$ Designed for input-dependent:

This allows the model to dynamically adjust the state update and forgetting speed according to the content of the current token; under common parameterization, $\Delta$ will affect the state decay and input injection intensity after discretization, and cannot be simply equated to a fixed “memory switch”.

hardware-aware parallel scan: directly performs a parallel prefix scan on the recursion of $h_t = \bar{A} h_{t-1} + \bar{B} x_t$, and the entire process is completed in the GPU SRAM to avoid frequent writing back to HBM. The GPU memory complexity is $O(N)$, and the training speed is comparable to Transformer.

Mamba block structure (each layer):

flowchart LR
    X["input x"] --> L1["linear projection × 2"]
    L1 --> |"main path"| CONV["1D Convolution<br/>(short range local perception)"]
    CONV --> ACT["SiLU activate"]
    ACT --> SSM["Selectivity SSM<br/>(Δ,B,C from input)"]
    SSM --> MUL["⊗ Gating"]
    L1 --> |"gated path"| MUL
    MUL --> L2["linear projection"]
    L2 --> Y["output y"]
    style SSM fill:#c4e1ff

Mamba 2(2024)

Mamba 2 (Dao & Gu, 2024) introduces State Space Duality (SSD), showing that Mamba’s selective SSM is a structured special case of linear attention. This enables optimization and implementation within a unified framework:

• Use matrix multiplication block (tensor cores) instead of scanning, faster than Mamba 1 2–8×
• Expand the state size to larger dimensions and improve model capacity

Transformer vs Mamba comparison

Representative work

Recent progress: combining recurrent memory and attention

Alternative architectures should distinguish pure SSMs, linear attention, and hybrid models. Finite-dimensional recurrent states help control decoding-cache size, but tasks differ in their requirements for exact retrieval, forgetting, and long-term memory. Hybrid designs retain some full attention while introducing cheaper recurrent layers.

Kimi Linear (2025) is one example: Kimi Delta Attention (KDA) extends Gated DeltaNet with fine-grained gating and is combined with MLA layers. It demonstrates an approach to balancing expressivity and long-context efficiency under a fixed training setup. Cache and throughput gains depend on the mixing ratio, sequence length, and implementation; the entire hybrid model should not be described as having exclusively linear computation. Original paper

11.5 Deep learning and LLMs

Deep learning (especially Transformer) is the technical foundation of large language models (LLM). The division of labor between this article and the LLM training review is as follows:

For specific engineering details of LLM pretraining/finetuning/inference, see:

Overview of LLM training technology

Covers topics such as Scaling Laws, distributed training, hybrid parallelism, KV Cache, GQA/MQA internal implementation, long context extrapolation, inference acceleration, etc.

12. Summary

There is only one main thread running through the whole text: machine learning three-step framework (Loss → Architecture → Optimization) × two core goals (Optimization vs Generalization). Each training technique accurately falls on a certain grid on this 3×2 coordinate system. For details, see the method classification summary table in Section 7.

Core conclusion:

Effective deep learning training requires matching the intervention to the problem, rather than adding more techniques. Inspect training loss first: if it does not decrease, address optimization before adding Dropout. Low training loss with high validation loss points to generalization, where data augmentation and regularization become relevant. This diagnostic principle is a foundation for effective experimentation and recurs throughout the article.

13. Primary sources for this update

These sources support the additions made on 2026-10-28. Method years generally refer to first public release, rather than conference publication.

• State-efficient methods and momentum improvements: Adafactor (2018), Lion (2023), Sophia (2023), and AdEMAMix (2024).
• Matrix optimization and geometric methods: Muon author’s explanation (2024) and implementation, Muon is Scalable for LLM Training / Moonlight (2025), SOAP (2024), Scion (ICML 2025), and NorMuon (ICML 2026).
• Recent 2026 preprints: Aurora: A Leverage-Aware Spectral Optimizer and AF-Muon: An AdamW-Free Muon Optimizer for Tied-Embedding Models.
• Learning rates and iterate averaging: Understanding Warmup-Stable-Decay Learning Rates (2024), The Road Less Scheduled (2024), and Schedule-Free’s official implementation and usage conventions.
• Low precision and attention kernels: DeepSeek-V3 Technical Report (2024), Scaling FP8 Training to Trillion-Token LLMs (2024), FlashAttention (2022), FlashAttention-2 (2023), FlashAttention-3 (2024), and FlashAttention-4 (2026).
• Preference learning, reasoning post-training, and hybrid architectures: DPO (2023), IPO (2023), KTO (2024), DeepSeekMath / GRPO (2024), DeepSeek-R1 (2025), DAPO (2025), and Kimi Linear (2025).