Transfer Learning — Technology Wiki

Overview

Direct Answer

Transfer learning is a machine learning methodology in which a model trained on a source domain or task is repurposed and fine-tuned for application to a target domain or task. This approach leverages learned representations—weights, feature hierarchies, and patterns—rather than training from random initialisation.

How It Works

A pre-trained model, typically trained on large-scale datasets, serves as a feature extractor. During the adaptation phase, either the final layers are retrained on new data whilst keeping earlier layers frozen, or all parameters undergo fine-tuning with a lower learning rate. This process preserves foundational knowledge whilst specialising the model to new target characteristics.

Why It Matters

Organisations deploy this technique to reduce computational cost, accelerate model development cycles, and achieve higher accuracy with limited labelled data. In resource-constrained settings, leveraging pre-trained models substantially lowers training time and infrastructure requirements whilst improving convergence speed and generalisation performance.

Common Applications

Computer vision applications utilise ImageNet-pretrained models for medical image analysis, object detection, and satellite imagery classification. Natural language processing commonly applies models trained on large corpora to domain-specific tasks including sentiment analysis, named entity recognition, and document classification.

Key Considerations

Domain mismatch between source and target can degrade performance if the learned representations are insufficiently similar. Practitioners must balance the trade-off between preserving pre-trained weights and adapting to target-specific characteristics, requiring careful selection of which layers to freeze and adjustment of hyperparameters.

Related in Advanced Methods

Semi-Supervised Learning

A learning approach that combines a small amount of labelled data with a large amount of unlabelled data during training.

Self-Supervised Learning

A learning paradigm where models generate their own supervisory signals from unlabelled data through pretext tasks.

Meta-Learning

Learning to learn — algorithms that improve their learning process by leveraging experience from multiple learning episodes.

Curriculum Learning

A training strategy that presents examples to a model in a meaningful order, typically from easy to hard.

Bagging

Bootstrap Aggregating — an ensemble method that trains multiple models on random subsets of data and averages their predictions.

Bandit Algorithm

An online learning algorithm that balances exploration of new options with exploitation of known good options to maximise reward.

More in Machine Learning

Matrix Factorisation

Unsupervised Learning

A technique that decomposes a matrix into constituent matrices, widely used in recommendation systems and dimensionality reduction.

Logistic Regression

Supervised Learning

A classification algorithm that models the probability of a binary outcome using a logistic function.

Feature Engineering

Feature Engineering & Selection

The process of using domain knowledge to create, select, and transform input variables to improve model performance.

Underfitting

Training Techniques

When a model is too simple to capture the underlying patterns in the data, resulting in poor performance on both training and test data.

Online Learning

MLOps & Production

A machine learning method where models are incrementally updated as new data arrives, rather than being trained in batch.

K-Nearest Neighbours

Supervised Learning

A simple algorithm that classifies data points based on the majority class of their k closest neighbours in feature space.

Decision Tree

Supervised Learning

A tree-structured model where internal nodes represent feature tests, branches represent outcomes, and leaves represent predictions.

Regularisation

Training Techniques

Techniques that add constraints or penalties to a model to prevent overfitting and improve generalisation to new data.