LinkedIn publication text
LoRA: how a 2021 idea became the standard way to customize AI models
Adapting a giant model to each task used to be expensive. On GPT-3 175B, every fine-tuned version took 350 GB, and full fine-tuning needed 1.2 TB of GPU memory.
In 2021, Edward J. Hu, Yelong Shen and six other Microsoft researchers proposed LoRA (Low-Rank Adaptation; arXiv:2106.09685, ICLR 2022). It freezes the model and trains only a small low-rank update, W₀ + BA.
📊 Result in the original paper: about 10,000× fewer trainable parameters, a 35 MB checkpoint instead of 350 GB, and no added inference latency after merging.
In this infographic I cover:
- 🟣 The theoretical roots (intrinsic dimension, 2018–2020) and the authors
- 🔵 The math and the engineering workflow: training, merging and multi-adapter serving (Punica, S-LoRA, vLLM)
- 🟢 A timeline of the variants: QLoRA, AdaLoRA, rsLoRA, VeRA, LoftQ, LongLoRA, DoRA, LoRA+ and PiSSA
- 🟡 Results and limits, from “LoRA Learns Less and Forgets Less” (2024) to “LoRA Without Regret” (2025)
- 🟠 A comparison with adapters, prompt tuning, BitFit, (IA)³, GaLore and RAG
- 🔴 Documented adoption: Azure OpenAI, Apple Intelligence, Tinker and Hugging Face
- 🛡️ Pros, cons and governance and security (GRC) considerations
✅ Bottom line: LoRA matches full fine-tuning in post-training and reinforcement learning when applied to all layers. It falls behind, however, when a model must absorb large amounts of new knowledge. The best combination is often LoRA for behavior and RAG for facts.
📚 Each slide explains why the topic matters and its positive and negative implications. It ends with a glossary and primary references, each one checked against its source.
Which of these topics matters most in your work today?