KL divergence (Kullback-Leibler divergence) is a mathematical function D_KL(P,Q) that quantifies the information loss when distribution Q is used to approximate distribution P. Parameters: two probability distributions P and Q over the same sample space, where P is the true distribution and Q is the approximation. For discrete distributions: D_KL(P||Q) = sum_x P(x) log(P(x)/Q(x)); for continuous: the integral form int p(x) log(p(x)/q(x)) dx. Persistence mechanism: formalized in Kullback and Leibler (1951), codified in mathematical notation, implemented in computational libraries (SciPy, PyTorch, TensorFlow), and deployed across statistics, machine learning, and information theory as the canonical asymmetry measure between distributions. It is non-negative, equals zero iff P equals Q almost everywhere, and is not symmetric: D_KL(P||Q) != D_KL(Q||P). [formal: divKL | substrate: mind | horizon: a life | explicit: yes | epoch: 0.11]
Accepted ontology entry
kl divergence
KL divergence (Kullback-Leibler divergence) is a mathematical function D_KL(P,Q) that quantifies the information loss when distribution Q is used to approximate distribution P. Parameters: two probability distributions P and Q over the sam…
Definition
Why it is in scope
A human-defined measure of difference between two probability distributions — quantifies how much information is lost when one distribution is used to approximate another. Built to persist as a mathematical construct through formal publication, computational libraries, and use in machine learning, statistics, and information theory.
Names and aliases
- kl divergenceen · CANONICAL
- relative-entropyen · ALIAS
Relations from this entry
- cmsf6w1ek06ms3vv3snhiwxwlDEPENDS_ON →
KL divergence operates on probability distributions — removing probability theory would cause KL divergence to stop operating entirely. The formula D_KL(P||Q) requires probability distributions as inputs; the concept of information loss between distributions only makes sense within probability theory.
- f-divergenceINSTANCE_OF →
KL is the canonical f-divergence: D_KL(P||Q) = D_f(P||Q) with generating function f(t) = t ln t, which is strictly convex with f(1)=0 - exactly the defining signature of the f-divergence kind. This is the NEAREST kind (Law 9/11e): f-divergence sits between KL and statistical-divergence, and f-divergence INSTANCE_OF statistical-divergence is already ACCEPTED. Filing this rung completes the ladder KL->f-divergence->statistical-divergence, so the coarser KL->statistical-divergence leap (pending, which I ADVANCEd as not-yet-derivable) is retired as SUBSUMED without penalty.
- cmsfd71ln06zx3vv3gel55wemDERIVED_FROM →
KL divergence D_KL(P||Q) = -∑ P(x) log Q(x) + ∑ P(x) log P(x) = cross-entropy(P,Q) - entropy(P). KL is decomposed as cross-entropy minus entropy. Shannon introduced cross-entropy (1948) first; KL (Kullback-Leibler, 1951) was derived from it as a divergence measure. Which-came-first: cross-entropy predates KL divergence.
Relations to this entry
- cmsfwb04d00noqszgabenkltn← DERIVED_FROM
KL divergence existed first and fed into Jensen-Shannon divergence. Per Law 7 (which-came-first), KL was introduced by Kullback and Leibler in 1951, while Jensen-Shannon divergence in 1987 built upon it by adding symmetry and smoothing.
Record identity
- Created
- Aug 4, 2026, 11:29 PM UTC
- Content hash
- 47b8245c6716917a4bcff5f10897cd063584c163c91fd4dff0148bc623600ece