Step by Step
F
Freeze the base model — no changes to original weights
LoRA keeps the entire original pretrained weight matrix W completely frozen, unchanged throughout fine-tuning.
Example: a massive pretrained language model's billions of original parameters remain completely untouched during LoRA fine-tuning.
L
Low-rank update — training a tiny fraction of parameters
Instead of updating the full weight matrix, LoRA adds a low-rank update ΔW = B × A, where B is d×r and A is r×k, with r chosen to be much smaller than d — meaning only A and B need to be trained, representing 100 to 10,000 times fewer parameters than the full matrix.
Example: instead of training billions of parameters in the full weight matrix, LoRA might only need to train a few million parameters in the much smaller low-rank matrices.
M
Merge back — zero added latency at inference
At inference time, the trained low-rank update can be merged directly back into the original weight matrix W, meaning there's no additional computational latency penalty compared to using the original model.
Example: after fine-tuning, the small B×A update is mathematically merged back into W, so the fine-tuned model runs exactly as fast as the original.
Q
QLoRA — combining 4-bit quantization with LoRA
QLoRA combines 4-bit quantization of the frozen base model with LoRA's low-rank fine-tuning, dramatically reducing memory requirements — enough to fine-tune a 65-billion-parameter model on a single consumer GPU, democratizing LLM fine-tuning.
Example: QLoRA making it possible for an individual researcher with a single consumer-grade GPU to fine-tune a massive 65B parameter model that would otherwise require enterprise-grade hardware.
Applied Walkthrough
1
A researcher wants to fine-tune a massive 65-billion-parameter language model, but only has access to a single consumer-grade GPU.
2
Fully fine-tuning all 65 billion parameters directly would require far more memory and compute than a single consumer GPU can provide.
3
Using QLoRA — combining 4-bit quantization of the frozen base model with LoRA's low-rank update training — dramatically reduces the memory footprint required.
4
This combination makes it feasible to fine-tune this massive model on a single consumer GPU, something that would have been completely impractical with traditional full fine-tuning.
Exam Application
Exams test whether you understand LoRA's core mechanism (freeze the base model, train only a small low-rank update, merge back for zero added latency) and whether you know QLoRA specifically adds 4-bit quantization on top of LoRA to further reduce memory requirements, enabling fine-tuning of very large models on modest hardware.
⚠ Common Trap
The most common trap is assuming LoRA fine-tuning adds extra inference latency, since it introduces new trainable parameters. In fact, the trained low-rank update gets merged directly back into the original weight matrix at inference time, resulting in zero added latency compared to the original model.
✓ Quick Self-Check
1. Does LoRA update the original pretrained weight matrix directly?
No — the original weight matrix stays completely frozen; LoRA trains a separate, much smaller low-rank update instead.
Tap to reveal / hide
2. Roughly how many fewer parameters does LoRA typically train compared to full fine-tuning?
100 to 10,000 times fewer parameters.
Tap to reveal / hide
3. Does LoRA fine-tuning add extra latency at inference time?
No — the trained low-rank update is merged back into the original weight matrix, resulting in zero added latency.
Tap to reveal / hide
4. What does QLoRA add on top of standard LoRA?
4-bit quantization of the frozen base model, further reducing memory requirements.
Tap to reveal / hide
5. What did QLoRA specifically enable, in terms of hardware accessibility?
Fine-tuning a 65-billion-parameter model on a single consumer GPU, democratizing LLM fine-tuning.
Tap to reveal / hide