Step by Step
A
Ask — pose a yes/no question about a feature
At each node, the tree asks a yes/no-style question about one specific feature, chosen specifically to best split the remaining data.
Example: "Is income greater than $50,000?" as the very first question asked in a loan-approval decision tree.
S
Split — divide the data to maximize information gain
The chosen question splits the data into subsets, with the specific question selected to maximize information gain — making the resulting groups as "pure" (containing mostly one class) as possible.
Example: splitting on income might separate applicants into a subgroup that's mostly approved and a subgroup that's mostly denied, which is a more informative split than a poorly-chosen question.
R
Repeat until pure — reaching the leaves
This process of asking and splitting repeats recursively down each branch until the resulting groups (leaves) are pure, or until a maximum depth limit is reached, at which point the leaf holds the final prediction.
Example: like playing 20 Questions — "Is it alive? Does it have legs? Does it bark?" — with each answer narrowing down the possibilities until you can confidently guess the answer.
Applied Walkthrough
1
A decision tree is being built to classify animals based on a series of yes/no questions about their characteristics.
2
The first question, "Does it have fur?", splits the data into two large, still-mixed groups.
3
The tree continues asking further questions — "Does it bark?", "Does it purr?" — each one further splitting the data, chosen specifically to maximize information gain at each step.
4
This continues until each final leaf contains an overwhelmingly pure group (like "dog" or "cat"), exactly mirroring how 20 Questions narrows down to a single confident answer through a sequence of well-chosen yes/no questions.
Exam Application
Exams test whether you understand that each split is chosen specifically to maximize information gain (making resulting groups as pure as possible) and whether you know decision trees are interpretable (easy to explain the reasoning behind a prediction) but prone to overfitting without techniques like pruning or a depth limit.
⚠ Common Trap
The most common trap is assuming decision trees always generalize well simply because they're interpretable. A single decision tree, if allowed to grow without a depth limit or pruning, will often overfit badly, memorizing very specific, narrow combinations of training data rather than learning generalizable patterns.
✓ Quick Self-Check
1. What criteria determines which question a decision tree asks at each split?
Whichever question maximizes information gain, making the resulting groups as pure as possible.
Tap to reveal / hide
2. What is a "leaf" in a decision tree?
A terminal node containing the tree's final prediction, reached once the data is pure enough or a depth limit is hit.
Tap to reveal / hide
3. Why is a decision tree compared to the game of 20 Questions?
Because it narrows down the possibilities through a sequence of yes/no questions, just like 20 Questions narrows down to a single answer.
Tap to reveal / hide
4. Are decision trees generally considered interpretable or a "black box"?
Interpretable — the sequence of questions leading to a prediction can be easily explained and traced.
Tap to reveal / hide
5. What problem can a single decision tree suffer from if allowed to grow without limits?
Overfitting, unless techniques like pruning or a maximum depth limit are applied.
Tap to reveal / hide