Knowledge distillation is a model compression technique in which a smaller 'student' neural network is trained to replicate the output probability distribution of a larger 'teacher' model. Parameters: (1) a pre-trained teacher model with established weights, (2) a smaller student architecture, (3) a temperature parameter T controlling the softness of the probability distribution, (4) a composite loss combining standard task loss on hard labels plus a KL-divergence loss between teacher and student softened outputs. Persistence: documented in peer-reviewed literature (Hinton et al., 2015), implemented in open-source frameworks (PyTorch, TensorFlow), and deployed in production inference pipelines where latency and compute constraints demand compact models. The technique operates as a protocol — a defined procedure for transferring learned representations from one model to another.\n\n[formal: distillatio | substrate: behavior | horizon: generations | explicit: yes | epoch: 0.01]
Accepted ontology entry
knowledge distillation
Knowledge distillation is a model compression technique in which a smaller 'student' neural network is trained to replicate the output probability distribution of a larger 'teacher' model. Parameters: (1) a pre-trained teacher model with e…
Definition
Why it is in scope
A human-made machine learning training technique in which a compact 'student' model learns to replicate the behavior of a larger 'teacher' model by minimizing the divergence between their output distributions. Created by Hinton et al. (2015) and documented across papers, codebases, and industry practice — it persists as a repeatable method for compressing model knowledge into smaller, faster deployments.
Names and aliases
- knowledge distillationen · CANONICAL
Relations from this entry
- cmrg0pc1x00e82a1nppcnwovjDEPENDS_ON →
Knowledge distillation operates by training a student neural network to mimic a teacher model's outputs. Remove neural networks and the technique cannot operate — it has no meaning without NN architectures. Present-tense dependency per Law 8.
- cmslcuob806rtnobpa1a6dlpbDEPENDS_ON →
Remove the teacher model and knowledge distillation cannot operate — the technique has no supervisory signal to replicate. Present-tense removal test passes: without a teacher, there is no distillation.
- cmsa8waf503477skqyquk9janINSTANCE_OF →
Knowledge distillation is a specific kind of training method in machine learning. A competent speaker would call it 'a training technique.' Nearest kind: training, not model compression (which is a goal, not a training method).
- cmsg3eea8012uqszga31dfeiwDERIVED_FROM →
Chronology test: transfer learning (experiments in adapting pre-trained models) predated knowledge distillation (Hinton et al., 2015). Knowledge distillation evolved from the practice of transferring learned representations — it is a specific technique within that broader lineage. Transfer learning existed as a concept and practice before distillation refined it.
- cmshsqfcc03c4ywh58twky2i9INSTANCE_OF →
Knowledge distillation is a specific kind of distillation in the ML sense — the transfer and purification of learned representations from one model to another. A competent speaker calls it 'a distillation technique.' Nearest kind: distillation.
- cmsldqgnp06v2nobp35goel8mDEPENDS_ON →
Knowledge distillation as a process requires a student model to receive and internalize the transferred knowledge — remove student models and knowledge distillation ceases to operate, not merely become unsayable. Present-tense removal test passes.
- cmsle0laa06w4nobp7rttnbliINSTANCE_OF →
Knowledge distillation IS a specific kind of representation learning — it transfers compressed internal representations from a teacher to a student model. A competent speaker calls it 'a form of representation learning.' Nearest kind: representation learning.
- cmsg3eea8012uqszga31dfeiwINSTANCE_OF →
Knowledge distillation is a specific kind of transfer learning — a student model learns from a teacher model to transfer compressed knowledge. A competent speaker would call it a transfer learning method. Files against nearest kind.
- cmsldf7d206tqnobp13puglliINSTANCE_OF →
knowledge distillation is a specific kind of model compression — per Law 9, a competent speaker would call knowledge distillation 'a model compression technique'. It compresses a large model into a smaller one by distilling knowledge.
Relations to this entry
- cmsldqgnp06v2nobp35goel8m← DEPENDS_ON
Student models are defined by their role in knowledge distillation. Remove knowledge distillation and the concept of student models ceases to operate — there is no student model outside of distillation. Present-tense removal test passes.
- cmslcuob806rtnobpa1a6dlpb← SERVES
Teacher models are maintained for the sake of knowledge distillation — their designed purpose is to produce outputs that a student model can learn from. SERVES arrow points servant→master per Law 8d.
Record identity
- Created
- Aug 9, 2026, 5:11 AM UTC
- Content hash
- 4f659a65d36817bce1b2b59488c06778ded6bc74a3d5c8b14804983ff976228b