Step by Step
P
Principal Components — the axes of maximum spread
PCA identifies directions (principal components) along which the data spreads out the most. PC1 captures the most variance; PC2 captures the next most, while being perpendicular to PC1; and so on.
Example: in a dataset of correlated financial metrics, PC1 might represent an overall "company size" direction capturing most of the variance across all the original features.
C
Compress without losing shape — projecting onto top components
Once the principal components are identified, the original data is projected onto just the top K components, discarding the rest, while still preserving most of the meaningful structure (variance) in the original data.
Example: projecting 500 original features onto just the top 10 principal components, which together still capture 95% of the original variance.
A
Applications — speed, overfitting reduction, and visualization
Reducing dimensionality this way speeds up training (fewer features to process), reduces overfitting risk (fewer parameters relative to data), and enables visualizing high-dimensional data in 2D or 3D.
Example: reducing a 50-feature dataset down to just its top 2 principal components specifically to create a 2D scatter plot visualization.
Applied Walkthrough
1
A dataset has 500 highly correlated features, making models slow to train and prone to overfitting.
2
PCA is applied, identifying the directions of maximum variance across these 500 correlated features.
3
The data is then projected onto just the top 10 principal components, which together happen to capture 95% of the original variance.
4
Training on these 10 components instead of the original 500 features speeds up training dramatically and reduces overfitting risk, while still preserving nearly all the meaningful structure in the data.
Exam Application
Exams test whether you understand PCA's core mechanism (finding axes of maximum spread, projecting data onto the top K of them) and its practical benefits (speed, reduced overfitting, visualization) — this is one of the most consistently tested dimensionality reduction topics across ML and algorithms courses.
⚠ Common Trap
The most common trap is assuming PCA works well without any preprocessing. Since PCA is based on variance, features with larger numeric scales will dominate the identified principal components unless all features are standardized first — a critical preprocessing step that's easy to overlook.
✓ Quick Self-Check
1. What does PC1 represent in PCA?
The single direction (axis) capturing the greatest amount of variance in the data.
Tap to reveal / hide
2. What is the relationship between PC1 and PC2?
PC2 captures the next greatest amount of variance, while being perpendicular (orthogonal) to PC1.
Tap to reveal / hide
3. Name three practical benefits of using PCA for dimensionality reduction.
Faster training, reduced overfitting risk, and enabling visualization in 2D or 3D.
Tap to reveal / hide
4. Give an example use case described for PCA in this lesson.
Reducing 500 features down to 10 principal components that capture 95% of the variance.
Tap to reveal / hide
5. What preprocessing step is critical before applying PCA?
Standardizing the data, since PCA is variance-based and large-scale features would otherwise dominate.
Tap to reveal / hide