Self-Attention — Technology Wiki

Overview

Direct Answer

Self-attention is a neural network mechanism that allows each position in a sequence to compute a weighted representation by attending to all other positions, including itself. It enables the model to dynamically learn which parts of the input are most relevant for processing each element, without relying on positional proximity or recurrence.

How It Works

The mechanism operates through three learnable projections—query, key, and value—that transform input sequences into corresponding representations. For each position, the query is compared against all keys using a scaled dot-product operation to produce attention weights, which are then applied to the values to create context-aware output vectors. This computation occurs in parallel across all sequence positions.

Why It Matters

Self-attention underpins transformer architectures that have become foundational to large language models and multimodal systems, delivering superior performance on sequential tasks whilst enabling efficient parallelisation during training. Organisations benefit from dramatically improved accuracy on language understanding, translation, and generation tasks, along with reduced computational overhead compared to recurrent alternatives for inference at scale.

Common Applications

This mechanism powers natural language processing applications including machine translation, text classification, and question-answering systems. It is also integral to vision transformers for image classification, multimodal models for cross-modal alignment, and time-series forecasting in financial and IoT contexts.

Key Considerations

Computational complexity scales quadratically with sequence length, creating bottlenecks for very long documents or high-resolution images. Attention patterns can also be difficult to interpret, and the mechanism requires sufficient training data to learn meaningful alignment patterns effectively.

Cross-References(1)

Deep Learning

Attention Mechanism

Related in Training & Optimisation

Multi-Head Attention

An attention mechanism that runs multiple attention operations in parallel, capturing different types of relationships.

Residual Network

A deep neural network architecture using skip connections that allow gradients to flow directly through layers, enabling very deep networks.

Layer Normalisation

A normalisation technique that normalises across the features of each individual sample rather than across the batch.

Dropout

A regularisation technique that randomly deactivates neurons during training to prevent co-adaptation and reduce overfitting.

Activation Function

A mathematical function applied to neural network outputs to introduce non-linearity, enabling the learning of complex patterns.

ReLU

Rectified Linear Unit — an activation function that outputs the input directly if positive, otherwise outputs zero.

Sigmoid Function

An activation function that maps input values to a range between 0 and 1, useful for binary classification outputs.

Softmax Function

An activation function that converts a vector of numbers into a probability distribution, commonly used in multi-class classification.

Positional Encoding

A technique that injects information about the position of tokens in a sequence into transformer architectures.

Gradient Clipping

A technique that caps gradient values during training to prevent the exploding gradient problem.

Mixed Precision Training

Training neural networks using both 16-bit and 32-bit floating-point arithmetic to speed up computation while maintaining accuracy.

Rotary Positional Encoding

A position encoding method that encodes absolute position with a rotation matrix and naturally incorporates relative position information into attention computations.

More in Deep Learning

Neural Network

Architectures

A computing system inspired by biological neural networks, consisting of interconnected nodes that process information in layers.

Representation Learning

Architectures

The automatic discovery of data representations needed for feature detection or classification from raw data.

State Space Model

Architectures

A sequence modelling architecture based on continuous-time dynamical systems that processes long sequences with linear complexity, offering an alternative to attention-based transformers.

Pooling Layer

Architectures

A neural network layer that reduces spatial dimensions by aggregating values, commonly using max or average operations.

Generative Adversarial Network

Generative Models

A framework where two neural networks compete — a generator creates synthetic data while a discriminator evaluates its authenticity.

Exploding Gradient

Architectures

A problem where gradients grow exponentially during backpropagation, causing unstable weight updates and training failure.

Contrastive Learning

Architectures

A self-supervised learning approach that trains models by comparing similar and dissimilar pairs of data representations.

Pre-Training

Language Models

The initial phase of training a deep learning model on a large unlabelled corpus using self-supervised objectives, establishing general-purpose representations for downstream adaptation.