Overview
Direct Answer
Model distillation is a compression technique in which a smaller, more efficient model—called the student—learns to approximate the predictions and internal representations of a larger, pre-trained model—called the teacher. The student model achieves comparable performance whilst requiring substantially fewer parameters and computational resources.
How It Works
The teacher model generates soft probability distributions (logits) over training data, which encode richer decision boundaries than hard labels alone. The student is trained on a combined loss function that minimises divergence from the teacher's soft outputs whilst maintaining accuracy on the original task. Temperature scaling adjusts the softness of these distributions, controlling the level of knowledge transfer and enabling the student to learn generalised patterns rather than memorising specific examples.
Why It Matters
Distillation enables deployment of high-performance models on resource-constrained devices, reducing latency, energy consumption, and infrastructure costs in production environments. This is critical for real-time applications such as mobile inference, edge computing, and large-scale serving where computational efficiency directly impacts operational expenses and user experience.
Common Applications
Applications include compressing language models for on-device natural language processing, accelerating computer vision models in autonomous systems, and optimising recommendation engines in e-commerce platforms. Financial institutions use distillation to deploy fraud detection models with lower latency, whilst healthcare organisations compress diagnostic models for integration into clinical decision-support systems.
Key Considerations
Knowledge transfer is not guaranteed; student models may fail to capture all nuances of teacher behaviour, particularly on out-of-distribution data. Determining optimal student architecture, temperature hyperparameters, and loss weighting between task accuracy and teacher mimicry requires substantial experimentation and validation.
More in Artificial Intelligence
AI Agent Orchestration
Infrastructure & OperationsThe coordination and management of multiple AI agents working together to accomplish complex tasks, routing subtasks between specialised agents based on capability and context.
AI Explainability
Safety & GovernanceThe ability to describe AI decision-making processes in human-understandable terms, enabling trust and regulatory compliance.
AI Benchmark
Evaluation & MetricsStandardised tests and datasets used to evaluate and compare the performance of AI models across specific tasks.
Symbolic AI
Foundations & TheoryAn approach to AI that uses human-readable symbols and rules to represent problems and derive solutions through logical reasoning.
Artificial Intelligence
Foundations & TheoryThe simulation of human intelligence processes by computer systems, including learning, reasoning, and self-correction.
Heuristic Search
Reasoning & PlanningProblem-solving techniques that use practical rules of thumb to find satisfactory solutions when exhaustive search is impractical.
TinyML
Evaluation & MetricsMachine learning techniques optimised to run on microcontrollers and extremely resource-constrained embedded devices.
AI Red Teaming
Safety & GovernanceThe systematic adversarial testing of AI systems to identify vulnerabilities, failure modes, harmful outputs, and safety risks before deployment.