Shannon entropy is a mathematical measure of the average uncertainty or information content in a random variable's probability distribution. For a discrete random variable X with outcomes x₁, x₂, ..., xₙ and probability mass function p(x), the entropy is defined as H(X) = -Σ p(x) log₂ p(x), where the sum ranges over all outcomes. The unit is bits when using base-2 logarithms. The formalism was introduced by Claude Shannon in his 1948 paper 'A Mathematical Theory of Communication' as the foundational quantity of information theory. It quantifies the expected number of bits required to encode symbols drawn independently from the distribution. Shannon entropy is maximized when all outcomes are equally likely (uniform distribution) and minimized (zero) when the outcome is deterministic. It underlies data compression limits, channel capacity theorems, and mutual information calculations. [formal: entropy | substrate: mind | horizon: a life | explicit: yes | epoch: 0.01]
Accepted ontology entry
shannon entropy
Shannon entropy is a mathematical measure of the average uncertainty or information content in a random variable's probability distribution. For a discrete random variable X with outcomes x₁, x₂, ..., xₙ and probability mass function p(x),…
Definition
Why it is in scope
A measure of the average uncertainty or information content in a random variable's probability distribution. Quantifies the expected number of bits needed to encode symbols drawn from that distribution. Human-made as a mathematical formalism — defined by Claude Shannon in 1948 as H(X) = -Σ p(x) log₂ p(x) for discrete distributions. Built to persist through information theory, data compression, and statistical inference.
Names and aliases
- shannon entropyen · CANONICAL
Relations from this entry
- cmsf6w1ek06ms3vv3snhiwxwlDEPENDS_ON →
Shannon entropy measures uncertainty within a probabilistic framework. Remove probability theory and the concept of entropy as expected information content loses its substrate — probability distributions cease to exist as formal objects. The formula operates on probability values defined by probability axioms.
Relations to this entry
- cmsfn3z6r001oqszgsol6nolu← DEPENDS_ON
Mutual information I(X;Y) = H(X)+H(Y)-H(X,Y) — defined entirely in terms of Shannon entropy. Remove entropy and mutual information has no definition.
- renyi-entropy← DERIVED_FROM
Rényi entropy (1961) was introduced as a one-parameter generalization of Shannon entropy (1948). Shannon entropy is the α→1 limit of Rényi's formula. Historical: Shannon existed first and fed into Rényi's generalization.
Record identity
- Created
- Aug 5, 2026, 10:15 AM UTC
- Content hash
- 9b98baaee7493e78a0a7ba4f72694020e8c6b48785c3d09728bd93ce775cdcc4