Overview
Direct Answer
Pipeline parallelism is a distributed training technique that partitions neural network layers across multiple devices and processes overlapping micro-batches through sequential stages to reduce idle time and maximise device utilisation. Unlike data parallelism, which replicates the full model across devices, this approach divides the model itself into stages that operate concurrently on different micro-batches.
How It Works
Each device holds a distinct set of consecutive layers, forming a pipeline stage. During forward propagation, micro-batch 1 advances through stage 1 while micro-batch 2 enters stage 1 and micro-batch 3 waits at the pipeline entrance. Backward propagation follows similarly, allowing devices to compute gradients while upstream stages process new data, thereby overlapping computation and communication to reduce bubble time—periods when devices remain idle waiting for dependencies.
Why It Matters
This approach enables training of extremely large models that exceed single-device memory capacity, directly reducing training time and hardware costs for organisations developing large language models and vision transformers. It addresses the memory bottleneck that prevents scaling beyond device VRAM limits, making feasible the training of multi-billion-parameter systems that would otherwise require prohibitively expensive hardware.
Common Applications
Pipeline parallelism is widely deployed in large-scale language model training by research institutions and cloud providers. It is essential for transformer-based architectures with 10+ billion parameters, particularly in natural language processing and multimodal AI development where models exceed individual GPU or TPU memory constraints.
Key Considerations
Pipeline bubble—idle device time between forward and backward passes—remains a fundamental efficiency loss; bubble fraction increases with deeper pipelines and smaller micro-batches. Practitioners must balance micro-batch size, pipeline depth, and gradient accumulation steps to optimise throughput whilst maintaining convergence behaviour and numerical stability.
Cross-References(3)
More in Deep Learning
Self-Attention
Training & OptimisationAn attention mechanism where each element in a sequence attends to all other elements to compute its representation.
Skip Connection
ArchitecturesA neural network shortcut that allows the output of one layer to bypass intermediate layers and be added to a later layer's output.
Multi-Head Attention
Training & OptimisationAn attention mechanism that runs multiple attention operations in parallel, capturing different types of relationships.
Activation Function
Training & OptimisationA mathematical function applied to neural network outputs to introduce non-linearity, enabling the learning of complex patterns.
Key-Value Cache
ArchitecturesAn optimisation in autoregressive transformer inference that stores previously computed key and value tensors to avoid redundant computation during sequential token generation.
Rotary Positional Encoding
Training & OptimisationA position encoding method that encodes absolute position with a rotation matrix and naturally incorporates relative position information into attention computations.
Gradient Clipping
Training & OptimisationA technique that caps gradient values during training to prevent the exploding gradient problem.
Dropout
Training & OptimisationA regularisation technique that randomly deactivates neurons during training to prevent co-adaptation and reduce overfitting.