Step by Step
T
Test feature — splitting data by a question
A decision tree splits data at each node by asking a question about a single feature, using Gini impurity or Information Gain to choose the best splitting feature.
Example: "Is income greater than $50,000?" as the first split in a loan-approval tree.
R
Recurse on subsets — repeat the splitting process
Each resulting subset of data is split again by the next best feature, recursively, building deeper and deeper branches.
Example: after splitting on income, the tree might next split the high-income subset by credit score.
E
End at leaf — a single tree, prone to overfitting alone
Splitting continues until reaching a "leaf" — a final prediction. A single decision tree taken this far tends to overfit badly, memorizing training data quirks.
Example: a very deep single tree might create a leaf for an oddly specific combination of features that only matches one or two training examples.
E
Ensemble to improve — Random Forest and XGBoost
Random Forest (bagging) grows many trees in parallel on random data and feature subsets, reducing variance. XGBoost (boosting) grows trees sequentially, each correcting the previous tree's errors, and dominates tabular data competitions.
Example: a Random Forest of 200 trees averages out the overfitting quirks of any single tree, producing a far more reliable overall prediction.
Applied Walkthrough
1
A single decision tree scores 99% on training data but only 62% on test data.
2
Ask: is this the classic sign of overfitting? Yes — the large training/test gap is a textbook overfitting signature.
3
The fix, rather than abandoning trees entirely, is to move to an ensemble method: Random Forest (bagging, parallel trees) or XGBoost (boosting, sequential trees).
4
Both ensemble approaches reduce the overfitting problem that plagues single decision trees, while keeping the interpretable, feature-splitting nature of tree-based models.
Exam Application
Exams frequently test whether you know tree-based methods don't require feature scaling (unlike K-NN or SVM), and whether you can distinguish bagging (parallel, reduces variance, Random Forest) from boosting (sequential, reduces bias, XGBoost). Also expect questions asking you to diagnose overfitting in a single-tree scenario and recommend the appropriate ensemble fix.
⚠ Common Trap
The most common trap is assuming Random Forest and XGBoost work the same way just because both use decision trees. Random Forest trains trees in PARALLEL and independently (bagging); XGBoost trains trees SEQUENTIALLY, with each new tree specifically targeting the previous tree's mistakes (boosting). Mixing these two mechanisms up is a frequent and heavily tested error.
✓ Quick Self-Check
1. What criteria does a decision tree use to choose the best feature to split on?
Gini impurity or Information Gain.
Tap to reveal / hide
2. Why do single decision trees tend to overfit?
Because they can keep splitting until leaves match very specific, narrow combinations of training examples, memorizing noise rather than general patterns.
Tap to reveal / hide
3. Does Random Forest train its trees in parallel or sequentially?
In parallel — each tree is trained independently on a random subset of data and features (bagging).
Tap to reveal / hide
4. Does XGBoost train its trees in parallel or sequentially?
Sequentially — each new tree specifically corrects the errors of the previous tree (boosting).
Tap to reveal / hide
5. Do tree-based methods require feature scaling like K-NN or SVM do?
No — tree-based methods split on individual feature values and don't need scaling.
Tap to reveal / hide