LoRA: Low-Rank Adaptation of Large Language Models

🤖 Plain-English Summary

An important paradigm of natural language processing consists of large-scale pre-training on general domain data and adaptation to particular tasks or domains. We also provide an empirical investigation into rank-deficiency in language model adaptation, which sheds light on the efficacy of LoRA.

🔑 Key Findings

• As we pre-train larger models, full fine-tuning, which retrains all model parameters, becomes less feasible.
• Using GPT-3 175B as an example -- deploying independent instances of fine-tuned models, each with 175B parameters, is prohibitively expensive.
• We propose Low-Rank Adaptation, or LoRA, which freezes the pre-trained model weights and injects trainable rank decomposition matrices into each layer of the Transformer architecture, greatly reducing the number of trainable parameters for downstream tasks.

🔗 Related Resources