Step by Step
K
Choosing K — how many neighbors get a vote
K determines how many of the closest training examples are consulted for the majority vote. K=1 means a single nearest neighbor decides alone; K=5 means the 5 closest points all vote.
Example: with K=1, a new point's classification is decided entirely by whichever single training point happens to be closest to it.
N
Nearest — measured by distance
"Nearest" is determined by a distance metric (typically Euclidean distance) between the new point and every point in the training set.
Example: calculating the straight-line distance between a new data point and every point in the training set to identify the K closest ones.
N
No training phase — lazy learning
K-NN doesn't build any internal model ahead of time — it simply stores all the training data and computes distances only at prediction time, earning it the nickname "lazy learning."
Example: unlike a decision tree that builds a structure in advance, K-NN's entire "training" step is just storing the data for later lookup.
Applied Walkthrough
1
A dataset is classified using K=1, and the resulting decision boundary between classes is found to be extremely jagged and irregular.
2
Ask: is a small K associated with a smoother or more jagged decision boundary? A more jagged boundary, since a very small K is highly sensitive to individual nearby points, including noisy ones.
3
Switching to a larger K, such as K=15, smooths out the decision boundary considerably, since more neighbors' votes average out individual noise.
4
This illustrates the core tradeoff: small K tends toward overfitting (jagged boundary), while large K tends toward underfitting (overly smooth boundary that may miss real local structure).
Exam Application
Exams test whether you understand the K tradeoff (small K = jagged, more overfit; large K = smoother, less overfit) and whether you recognize K-NN as a "lazy learner" with no real training phase, unlike algorithms that build an internal model ahead of time.
⚠ Common Trap
The most common trap is assuming a larger K is always better since it seems more "democratic." An excessively large K can smooth the decision boundary so much that it starts ignoring genuinely meaningful local structure in the data, leading to underfitting rather than improved accuracy.
✓ Quick Self-Check
1. What does K represent in K-Nearest Neighbors?
The number of closest training examples consulted to vote on a new point's classification.
Tap to reveal / hide
2. What kind of decision boundary does a small K (like K=1) tend to produce?
A jagged, irregular boundary that's more prone to overfitting.
Tap to reveal / hide
3. What kind of decision boundary does a large K tend to produce?
A smoother boundary, less prone to overfitting but potentially prone to underfitting if K is too large.
Tap to reveal / hide
4. Why is K-NN called a "lazy learner"?
Because it has no real training phase — it simply stores the training data and computes distances only at prediction time.
Tap to reveal / hide
5. What distance metric is typically used to determine a new point's nearest neighbors?
Euclidean distance (most commonly).
Tap to reveal / hide