Cross-entropy is a measure from information theory that quantifies the divergence between two probability distributions p and q over the same sample space. Formally defined as H(p,q) = -Σ p(x) log q(x) for discrete distributions or -∫ p(x) log q(x) dx for continuous ones, it represents the expected number of bits needed to encode events from distribution p using a code optimized for distribution q. The concept was introduced by Claude Shannon in 1948 as part of his foundational work on information theory, and has since become central to machine learning as a loss function for training probabilistic classifiers. Parameters: two probability distributions over identical support; base of logarithm determines units (bits for base 2, nats for base e). Persistence mechanism: mathematical formula encoded in notation, taught in information theory and statistics curricula, implemented in software libraries.
Accepted ontology entry
cross-entropy
Cross-entropy is a measure from information theory that quantifies the divergence between two probability distributions p and q over the same sample space. Formally defined as H(p,q) = -Σ p(x) log q(x) for discrete distributions or -∫ p(x)…
Definition
Why it is in scope
A human-made mathematical function from information theory that quantifies the difference between two probability distributions. Built to persist through formal notation and mathematical teaching — it is the expected number of bits needed to identify events from one distribution when using a code optimized for another.
Names and aliases
- cross-entropyen · CANONICAL
Relations from this entry
- cmreawf22000vg8vu90kto3goDERIVED_FROM →
Information theory was introduced by Shannon in 1948. Cross-entropy was formulated within that framework — information theory existed first and fed cross-entropy into being. Which came first? Information theory.
- cmsf6w1ek06ms3vv3snhiwxwlDERIVED_FROM →
Probability theory existed first and fed into cross-entropy. Cross-entropy is a measure from information theory whose mathematical form H(p,q) = -sum p(x)log q(x) directly uses probability distributions. Which came first? Probability theory — it predates information theory by centuries.
- cmsftujnv00f0qszgpolboiajDERIVED_FROM →
Cross-entropy DERIVED_FROM information theory — it was derived from Shannon's entropy concept, extending the information-theoretic measure of uncertainty to the divergence between two probability distributions (Law 7). Information theory existed first and provided the foundation.
- cmsfrqasq00aqqszg1stkbizqDERIVED_FROM →
Cross-entropy H(p,q) = -Σ p(x) log q(x) is the extension of Shannon entropy H(p) = -Σ p(x) log p(x) to two distributions. The entropy concept (Shannon 1948) existed first and cross-entropy was derived from it — the cross-entropy formula is literally the negative expected log of q, while entropy is the same form with p=q. Entropy is the more fundamental construct that cross-entropy derives from.
- cmrmi0v8g00bwd1nlrx23qz6gSERVES →
Cross-entropy is used as the standard loss function for classification tasks, especially multi-class classification with softmax outputs. It is built and maintained for the sake of classification — quantifying the difference between predicted probability distributions and true class labels. Law 8d: servant (cross-entropy) points at master (classification).
Relations to this entry
- kullback-leibler-divergence← DERIVED_FROM
KL-divergence D_KL(P||Q) = -sum P(x)log Q(x) + sum P(x)log P(x) = cross-entropy(P,Q) - entropy(P). KL is literally decomposed into cross-entropy minus entropy terms. Cross-entropy was introduced by Shannon (1948) within information theory; KL-divergence (1951) was derived from it as a divergence measure. Cross-entropy is the more fundamental construct.
- relative-entropy← DERIVED_FROM
KL divergence D_KL(P||Q) = -∑ P(x) log Q(x) + ∑ P(x) log P(x) = cross-entropy(P,Q) - entropy(P). KL is decomposed as cross-entropy minus entropy. Shannon introduced cross-entropy (1948) first; KL (Kullback-Leibler, 1951) was derived from it as a divergence measure. Which-came-first: cross-entropy predates KL divergence.
Record identity
- Created
- Aug 5, 2026, 12:44 AM UTC
- Content hash
- d09f71a05abaa2534c154c81cba8906c2e09bf850172a4876eea490d1e6d6dee