T
Training set — where the model actually learns
The largest portion of the data, used directly to adjust the model's weights and parameters. The model sees this data repeatedly during training and learns patterns from it.
Example: 70% of a labeled dataset might be set aside purely for the model to learn from during the training loop.
V
Validation set — used to tune, never to train
A separate slice of data used during development to tune hyperparameters (like learning rate or model complexity) and to check progress — the model never directly trains on this data, but it does get used repeatedly for tuning decisions.
Example: trying three different learning rates and checking validation accuracy for each to decide which one to keep, without ever letting the model train directly on validation examples.
T
Test set — locked away, touched only once
A completely separate slice of data, set aside and untouched until the very end of the project, used exactly once to measure real-world performance honestly.
Example: after all tuning decisions are finalized using the validation set, the test set is used a single time to report the model's final, honest accuracy.
Applied Walkthrough
1
A student tunes their model's hyperparameters, checking accuracy against their test set repeatedly to decide which settings work best.
2
Ask: is the test set supposed to be used for tuning decisions? No — tuning should only ever use the validation set.
3
By repeatedly checking against the test set during tuning, the student has effectively turned it into a second validation set — this is data leakage.
4
The reported "test accuracy" is now falsely inflated, because the test set influenced the tuning process instead of measuring genuinely unseen performance — the final number can no longer be trusted as an honest measure of real-world performance.
Exam Application
This is one of the most heavily tested practical concepts in any ML course, because it directly determines whether a reported accuracy number can be trusted. Exam questions often describe a workflow and ask you to spot exactly where the test set was touched more than once, or where the validation and test sets got mixed up — spotting that mistake is the entire point of the question.
⚠ Common Trap
The single most common and most heavily tested trap: touching the test set more than once, or using it during any tuning decision. The moment the test set influences any decision about the model (hyperparameters, architecture choices, feature selection) before the final evaluation, it stops being a valid measure of real-world performance — this is called data leakage, and it causes falsely inflated, untrustworthy scores.
✓ Quick Self-Check
1. Which data set does the model directly learn from during training?
The training set.
Tap to reveal / hide
2. Which data set is used to tune hyperparameters, without ever training the model directly on it?
The validation set.
Tap to reveal / hide
3. How many times should the test set ideally be used?
Only once — at the very end, to report final, honest real-world performance.
Tap to reveal / hide
4. What is "data leakage" in this context?
When information from the validation or test set improperly influences training or tuning decisions, causing falsely inflated performance scores.
Tap to reveal / hide
5. If a student repeatedly checks their model against the test set while tuning hyperparameters, what problem does this cause?
Data leakage — the test set has effectively become a second validation set, and the final reported accuracy can no longer be trusted as a genuine measure of unseen performance.
Tap to reveal / hide