Overview
Direct Answer
Data parallelism is a distributed training approach in which an identical model is replicated across multiple devices, each processing different subsets of training data in parallel, with gradient updates synchronised across all replicas after each iteration. This strategy enables significant acceleration of training for large datasets without modifying the model architecture.
How It Works
Each device holds a complete copy of the model and processes a distinct batch of training examples independently. After the forward pass and backpropagation, gradients computed on each device are aggregated (typically via averaging) through a synchronisation mechanism such as all-reduce. The synchronised gradients are then applied uniformly to update model weights across all replicas before the next iteration begins.
Why It Matters
Organisations training large-scale models benefit from reduced time-to-convergence, enabling faster experimentation cycles and reduced computational cost per training run. This approach scales nearly linearly with device count for large batch sizes, making it economically viable to train models on datasets that would be prohibitively slow on single-device setups.
Common Applications
Computer vision model training on image classification datasets, natural language processing tasks such as large transformer model pretraining, and recommendation system training on e-commerce platforms routinely employ this strategy to reduce wall-clock training time from weeks to days.
Key Considerations
Communication overhead between devices can become a bottleneck at scale, particularly with slower interconnects or very frequent synchronisation. Effective batch size increases with the number of devices, which may require adjusted learning rates and can affect model convergence behaviour and final accuracy if not compensated appropriately.
Cross-References(1)
More in Deep Learning
Self-Attention
Training & OptimisationAn attention mechanism where each element in a sequence attends to all other elements to compute its representation.
Positional Encoding
Training & OptimisationA technique that injects information about the position of tokens in a sequence into transformer architectures.
Exploding Gradient
ArchitecturesA problem where gradients grow exponentially during backpropagation, causing unstable weight updates and training failure.
Contrastive Learning
ArchitecturesA self-supervised learning approach that trains models by comparing similar and dissimilar pairs of data representations.
Activation Function
Training & OptimisationA mathematical function applied to neural network outputs to introduce non-linearity, enabling the learning of complex patterns.
Multi-Head Attention
Training & OptimisationAn attention mechanism that runs multiple attention operations in parallel, capturing different types of relationships.
Weight Decay
ArchitecturesA regularisation technique that penalises large model weights during training by adding a fraction of the weight magnitude to the loss function, preventing overfitting.
Gradient Checkpointing
ArchitecturesA memory optimisation that trades computation for memory by recomputing intermediate activations during the backward pass instead of storing them all during the forward pass.