Overview
Direct Answer
A skip connection is an architectural pattern that creates a direct pathway for the output of an earlier layer to be added element-wise to the output of a deeper layer, bypassing intermediate layers entirely. This mechanism enables networks to learn both residual transformations and identity mappings simultaneously.
How It Works
During forward propagation, the activation tensor from layer n is added directly to the output of layer n+k, where k represents the number of skipped layers. Backpropagation then routes gradients through both the skip pathway and the standard computational path, creating multiple gradient flow routes. This dual-path architecture reduces the vanishing gradient problem by ensuring that gradients maintain sufficient magnitude even in very deep networks.
Why It Matters
Skip connections substantially improve training stability and convergence speed in networks exceeding 50+ layers, directly reducing computational costs and wall-clock training time. They enable organisations to train significantly deeper models that achieve superior accuracy on vision and sequence tasks whilst remaining practically trainable on standard hardware infrastructure.
Common Applications
Residual networks (ResNets) in image classification and object detection; transformer architectures in natural language processing and large language models; U-Net style encoders in medical image segmentation; and very deep convolutional networks in autonomous vehicle perception systems.
Key Considerations
Skip connections require compatible tensor dimensions between source and target layers; dimensional mismatches necessitate learnable projection layers that increase computational overhead. The benefits diminish in shallow networks and recurrent architectures where gradient flow is naturally less problematic.
Cross-References(1)
Referenced By1 term mentions Skip Connection
Other entries in the wiki whose definition references Skip Connection — useful for understanding how this concept connects across Deep Learning and adjacent domains.
More in Deep Learning
Graph Neural Network
ArchitecturesA neural network designed to operate on graph-structured data, learning representations of nodes, edges, and entire graphs.
Contrastive Learning
ArchitecturesA self-supervised learning approach that trains models by comparing similar and dissimilar pairs of data representations.
Fine-Tuning
Language ModelsThe process of adapting a pre-trained model to a specific task by continuing training on a smaller task-specific dataset, transferring learned representations to new domains.
Positional Encoding
Training & OptimisationA technique that injects information about the position of tokens in a sequence into transformer architectures.
Residual Connection
Training & OptimisationA skip connection that adds a layer's input directly to its output, enabling gradient flow through deep networks and allowing training of architectures with hundreds of layers.
Mamba Architecture
ArchitecturesA selective state space model that achieves transformer-level performance with linear-time complexity by incorporating input-dependent selection mechanisms into the recurrence.
Key-Value Cache
ArchitecturesAn optimisation in autoregressive transformer inference that stores previously computed key and value tensors to avoid redundant computation during sequential token generation.
Parameter-Efficient Fine-Tuning
Language ModelsMethods for adapting large pretrained models to new tasks by only updating a small fraction of their parameters.