Step by Step
G
Greedy decoding — always the top token
Always picks the single highest-probability token at each step — deterministic and fast, but tends to produce repetitive, less creative output.
Example: greedy decoding might repeatedly generate the same safe, predictable phrase whenever it comes up, since it always picks the statistically most likely next token.
S
Sampling — randomly drawing from the distribution
Randomly samples the next token from the full probability distribution rather than always picking the top choice, producing more diverse output but risking occasional incoherence.
Example: pure random sampling might occasionally pick a low-probability, slightly odd word choice, adding variety but sometimes at the cost of coherence.
T
Temperature — controlling how sharp or flat the distribution is
Temperature below 1.0 sharpens the probability distribution (more conservative, predictable output); temperature above 1.0 flattens it (more creative, varied, but riskier output).
Example: a temperature of 0.3 produces safe, predictable text, while a temperature of 1.2 produces much more varied, surprising text.
P
Top-P (nucleus sampling) — the most widely used approach
Samples from the smallest possible set of tokens whose cumulative probability exceeds P, dynamically adapting to the shape of the probability distribution at each step — this is the most widely used decoding strategy in production LLM applications today.
Example: with P=0.9, the model considers only the smallest group of top tokens whose combined probability adds up to at least 90%, ignoring the long tail of very unlikely tokens.
Applied Walkthrough
1
A chatbot using pure greedy decoding produces noticeably repetitive, predictable responses across many conversations.
2
Switching to pure random sampling introduces more variety, but occasionally produces slightly incoherent or odd word choices.
3
Switching instead to Top-P (nucleus) sampling with P=0.9 dynamically considers only the smallest set of most likely tokens whose combined probability reaches 90%, adapting automatically to how confident or uncertain the model is at each step.
4
This adaptive behavior is exactly why nucleus sampling has become the standard choice in production LLM applications — it balances diversity and coherence better than either pure greedy or pure random sampling alone.
Exam Application
Exams test whether you can distinguish these decoding strategies (greedy, sampling, temperature, top-K, top-P) and specifically whether you know top-P (nucleus sampling) is the most widely used approach in production, due to its ability to dynamically adapt to the shape of the probability distribution.
⚠ Common Trap
The most common trap is confusing top-K with top-P. Top-K always samples from a fixed number of top tokens regardless of the distribution's shape; top-P (nucleus) dynamically adjusts how many tokens are considered based on the cumulative probability threshold, adapting to whether the model is very confident (few tokens needed) or very uncertain (many tokens needed) at each step.
✓ Quick Self-Check
1. What does greedy decoding always do?
Picks the single highest-probability token at each step, producing deterministic but often repetitive output.
Tap to reveal / hide
2. What does temperature above 1.0 do to the probability distribution?
Flattens it, producing more creative but riskier, more varied output.
Tap to reveal / hide
3. What is the key difference between top-K and top-P sampling?
Top-K samples from a fixed number of top tokens regardless of distribution shape; top-P dynamically adjusts how many tokens are considered based on cumulative probability.
Tap to reveal / hide
4. What does top-P (nucleus) sampling specifically select?
The smallest set of tokens whose cumulative probability exceeds a threshold P.
Tap to reveal / hide
5. Which decoding strategy is most widely used in production LLM applications?
Top-P (nucleus) sampling.
Tap to reveal / hide