Kullback–Leibler divergence is a measure of how one probability distribution P diverges from a reference distribution Q. For discrete distributions: D_KL(P || Q) = Σₓ P(x) log(P(x)/Q(x)); for continuous distributions, the sum becomes ∫ p(x) log(p(x)/q(x)) dx. It quantifies the expected excess number of bits when using Q to encode data drawn from P. KL-divergence is non-negative (D_KL ≥ 0, equality iff P = Q almost everywhere) but is not symmetric and does not satisfy the triangle inequality — making it a divergence, not a metric. It arises naturally as the Bregman divergence of the log-partition function in exponential families, as the rate function in Sanov's theorem of large deviations, and as the objective function in variational inference (where KL(q||p) measures the gap between approximate and true posterior). Introduced by Solomon Kullback and Richard Leibler (1951), it is foundational to information theory, machine learning, and statistical physics. [formal: divergentia-kullback-leibler | substrate: mind | horizon: a life | explicit: yes | epoch: 0.01]
Accepted ontology entry
kullback-leibler-divergence
Kullback–Leibler divergence is a measure of how one probability distribution P diverges from a reference distribution Q. For discrete distributions: D_KL(P || Q) = Σₓ P(x) log(P(x)/Q(x)); for continuous distributions, the sum becomes ∫ p(x…
Definition
Why it is in scope
A human-made mathematical concept in information theory and statistics: a measure of how one probability distribution diverges from a reference distribution. Built to persist in information theory, machine learning, Bayesian inference, and large deviation theory.
Names and aliases
- kullback-leibler-divergenceen · CANONICAL
Relations from this entry
- cmsfd71ln06zx3vv3gel55wemDERIVED_FROM →
KL-divergence D_KL(P||Q) = -sum P(x)log Q(x) + sum P(x)log P(x) = cross-entropy(P,Q) - entropy(P). KL is literally decomposed into cross-entropy minus entropy terms. Cross-entropy was introduced by Shannon (1948) within information theory; KL-divergence (1951) was derived from it as a divergence measure. Cross-entropy is the more fundamental construct.
- f-divergenceINSTANCE_OF →
KL-divergence is a specific f-divergence with generating function f(t) = t log(t). A competent speaker would call KL-divergence an f-divergence. This is the nearest kind — f-divergence is more specific than the broader statistical-divergence.
- cmsfrqasq00aqqszg1stkbizqDERIVED_FROM →
KL-divergence D_KL(P||Q) = Σ P(x) log(P(x)/Q(x)) = -Σ P(x) log Q(x) + Σ P(x) log P(x) = cross-entropy(P,Q) - entropy(P). The KL formula is literally decomposed into Shannon entropy terms. It was derived from the entropy concept by Kullback and Leibler (1951). Entropy (Shannon 1948) predates and feeds into KL-divergence.
Relations to this entry
- evidence-lower-bound← DEPENDS_ON
The ELBO's defining identity log p(x) = L(q) + KL(q||p) means L(q) = log p(x) - KL(q||p). Remove KL-divergence and the ELBO operational definition collapses. The variational inference machinery optimizing q to minimize the KL gap ceases to function. Present-tense operational necessity.
- cmsfn3z6r001oqszgsol6nolu← DEPENDS_ON
Mutual information I(X;Y) = D_KL(p(x,y) || p(x)p(y)) — its definition is literally the KL-divergence between the joint distribution and the product of marginals. Remove KL-divergence and mutual-information has no definition. Constitutive present-tense dependency: mutual-information operates through KL-divergence as its computational mechanism (Law 8b/8c).
Record identity
- Created
- Sep 3, 2026, 6:41 PM UTC
- Content hash
- 2b5daa12ea84119ec37bb3cc302cce96956ef4c5c9760cc748816fa7c5d2e0d6