Step by Step
D
DQN — a neural network approximating Q-values
Deep Q-Networks use a neural network to approximate the Q-value (expected future reward) for every possible action in a given state, allowing reinforcement learning to scale to large or continuous state spaces where a simple lookup table would be infeasible.
Example: using a neural network to estimate the expected future reward of each possible move in a video game with an enormous number of possible screen states.
E
Experience Replay — breaking temporal correlations
DQN stores past experiences (transitions) in a memory buffer and samples from them randomly during training, rather than training only on the most recent, temporally correlated experiences — this breaks harmful correlations and stabilizes learning.
Example: randomly sampling from a large buffer of past game states and outcomes, rather than only training on the most recent few moves, which would otherwise be highly correlated with each other.
T
Target Network — a frozen reference for stability
A separate, periodically-updated "target" network provides stable Q-value targets during training, rather than using the constantly-changing main network for both predictions and targets, which would otherwise destabilize training.
Example: freezing a copy of the network for a while to generate stable targets, only updating that frozen copy periodically, rather than chasing a constantly moving target.
P
PPO — the dominant modern algorithm, used in RLHF
Proximal Policy Optimization clips policy updates to prevent overly large, destructive changes in a single training step, making it the most widely used Deep RL algorithm today — and specifically the algorithm used in RLHF for aligning models like ChatGPT and Claude.
Example: PPO carefully limiting how much the model's behavior can shift in a single training update, preventing a single bad update from catastrophically damaging previously learned good behavior.
Applied Walkthrough
1
A team wants to align a large language model's behavior using reinforcement learning based on human preference rankings.
2
Ask: which Deep RL algorithm is the standard choice for this specific task? PPO, since it's the dominant algorithm specifically used for RLHF in systems like ChatGPT and Claude.
3
PPO's careful clipping of policy updates helps ensure the language model's behavior doesn't change too drastically or destructively in any single training step.
4
This stability is exactly why PPO has become the standard choice not just for RLHF, but across many Deep RL applications more broadly.
Exam Application
Exams test whether you understand DQN's two key stabilizing innovations (experience replay and target networks) and whether you know PPO as the dominant modern Deep RL algorithm, specifically connected to its role in RLHF for aligning large language models.
⚠ Common Trap
The most common trap is assuming DQN alone (without experience replay and target networks) would train stably. Both of these specific innovations were necessary additions to make DQN training stable — without them, DQN training is prone to serious instability from correlated experiences and constantly shifting targets.
✓ Quick Self-Check
1. What problem does Experience Replay solve?
It breaks harmful temporal correlations by randomly sampling from a memory buffer of past experiences, rather than training only on the most recent, correlated experiences.
Tap to reveal / hide
2. What problem does the Target Network solve?
It stabilizes training by providing a stable, periodically-updated reference for Q-value targets, rather than chasing a constantly shifting target.
Tap to reveal / hide
3. What does PPO stand for?
Proximal Policy Optimization.
Tap to reveal / hide
4. What does PPO specifically do to stabilize training?
Clips policy updates to prevent overly large, destructive changes in a single training step.
Tap to reveal / hide
5. What real-world application is PPO specifically known for, related to LLMs?
RLHF (Reinforcement Learning from Human Feedback), used to align models like ChatGPT and Claude.
Tap to reveal / hide