Overview
Direct Answer
LoRA (Low-Rank Adaptation) is a parameter-efficient fine-tuning technique that trains small, low-rank matrices inserted alongside frozen pretrained model weights, rather than updating all parameters. This approach dramatically reduces the number of trainable parameters whilst maintaining model performance.
How It Works
LoRA decomposes weight updates into products of two smaller matrices with reduced rank dimensions. During fine-tuning, only these low-rank decomposition matrices are optimised whilst the original pretrained weights remain fixed. The adapted weights are computed as the sum of the frozen weights and the product of the trained low-rank matrices, scaled by a learning rate factor.
Why It Matters
This technique reduces memory consumption and computational cost by 10-100 times compared to full fine-tuning, making large model adaptation feasible on modest hardware. Organisations can now customise foundation models for specific tasks without prohibitive infrastructure investment or extended training timelines.
Common Applications
LoRA is widely deployed in adapting large language models for domain-specific tasks, personalising image generation models for particular artistic styles, and enabling efficient task-specific variants of models like LLaMA and Stable Diffusion. Financial institutions and healthcare organisations use it to specialise models whilst maintaining compliance boundaries.
Key Considerations
The technique introduces a rank hyperparameter that requires tuning; ranks that are too low may limit adaptation capability, whilst higher ranks diminish parameter efficiency gains. LoRA performs best when fine-tuning data resembles pretraining distribution; extreme distribution shifts may still require partial weight unfreezing.
Cross-References(2)
More in Deep Learning
Gradient Checkpointing
ArchitecturesA memory optimisation that trades computation for memory by recomputing intermediate activations during the backward pass instead of storing them all during the forward pass.
Fully Connected Layer
ArchitecturesA neural network layer where every neuron is connected to every neuron in the adjacent layers.
Mamba Architecture
ArchitecturesA selective state space model that achieves transformer-level performance with linear-time complexity by incorporating input-dependent selection mechanisms into the recurrence.
Dropout
Training & OptimisationA regularisation technique that randomly deactivates neurons during training to prevent co-adaptation and reduce overfitting.
Pretraining
ArchitecturesTraining a model on a large general dataset before fine-tuning it on a specific downstream task.
Transformer
ArchitecturesA neural network architecture based entirely on attention mechanisms, eliminating recurrence and enabling parallel processing of sequences.
Weight Decay
ArchitecturesA regularisation technique that penalises large model weights during training by adding a fraction of the weight magnitude to the loss function, preventing overfitting.
Residual Network
Training & OptimisationA deep neural network architecture using skip connections that allow gradients to flow directly through layers, enabling very deep networks.