Step by Step
1
Split into K equal folds
The dataset is divided into K roughly equal-sized folds, commonly K=5 or K=10.
Example: with K=5, a dataset of 1,000 examples gets split into five folds of 200 examples each.
2
Rotate: train on K-1, test on 1
In each round, one fold serves as the test fold while the remaining K-1 folds train the model, rotating through all K folds so each one gets tested exactly once.
Example: in round 1, fold 1 is the test fold and folds 2-5 train the model; in round 2, fold 2 becomes the test fold, and so on.
3
Average across all K rounds
After all K rounds, the K performance scores are averaged to produce a more reliable overall estimate than any single train/test split would provide.
Example: averaging five accuracy scores from five rounds into one overall mean accuracy estimate.
⚠
The golden rule — never touch the true test set except at the very end
Cross-validation is used during development and tuning, but a genuinely separate, held-out test set must be reserved and used ONLY for the final evaluation — never for any tuning decisions along the way.
Example: using cross-validation on the training data to tune hyperparameters, then evaluating the final, fully-tuned model exactly once on the untouched test set.
Applied Walkthrough
1
A team repeatedly checks their model's performance against their designated test set while trying out different hyperparameter combinations.
2
Ask: does this violate the golden rule? Yes — using the test set for any tuning decisions, even repeatedly checking it, effectively turns it into a second validation set.
3
This means their final reported "test accuracy" is no longer a trustworthy, honest measure of real-world performance, since the test set has already influenced their tuning choices.
4
The correct approach: use K-fold cross-validation on the training data alone for all tuning decisions, and only touch the true test set once, at the very end, for final evaluation.
Exam Application
Exams heavily test the golden rule that the test set must be touched only once, at the very end — using it for any tuning decision constitutes a serious data leakage-style error. Also expect questions on stratified K-fold (ensuring each fold has the same class distribution, important for imbalanced data) and the rule that time series data must always be split chronologically, never randomly.
⚠ Common Trap
The most common trap is treating the test set as a convenient way to "check progress" during model development. Any use of the test set beyond the single final evaluation compromises its validity as an honest measure of real-world performance — all tuning must happen using cross-validation or a separate validation set instead.
✓ Quick Self-Check
1. What does K-fold cross-validation do differently from a single train/test split?
It rotates through K different train/test splits and averages the results, producing a more reliable performance estimate.
Tap to reveal / hide
2. What is the golden rule regarding the test set?
Never use it for anything except the final evaluation — no tuning decisions should ever be based on test set performance.
Tap to reveal / hide
3. What does stratified K-fold ensure, and why does it matter?
It ensures each fold has the same class distribution as the overall dataset, which is important for imbalanced data.
Tap to reveal / hide
4. How should time series data be split for cross-validation?
Always chronologically — never randomly, to avoid leaking future information into training.
Tap to reveal / hide
5. What happens if you repeatedly check performance against the test set while tuning hyperparameters?
The test set effectively becomes a second validation set, and the final reported performance can no longer be trusted as an honest measure.
Tap to reveal / hide