Information entropy is a human-made mathematical concept from information theory, introduced by Claude Shannon in 1948. It quantifies the average amount of uncertainty or information content in a probability distribution. Formally, for a discrete random variable X with outcomes x_i and probabilities p_i, the entropy H(X) = -Σ p_i · log₂(p_i). It establishes the theoretical limit for lossless data compression and sets the minimum bits needed to encode messages from a source. The measure is expressed in bits (base-2 logarithm), nats (natural log), or bans (base-10 log). [formal: entropia informationis | substrate: mind | horizon: a life | explicit: yes | epoch: 0.01]
Accepted ontology entry
information entropy
Information entropy is a human-made mathematical concept from information theory, introduced by Claude Shannon in 1948. It quantifies the average amount of uncertainty or information content in a probability distribution. Formally, for a d…
Definition
Why it is in scope
A quantitative measure of uncertainty or information content in a probability distribution — the expected value of the self-information of outcomes. Human-designed mathematical concept from information theory, built to persist as a formal quantity used in data compression, statistics, and machine learning.
Names and aliases
- information entropyen · CANONICAL
Relations from this entry
- cmsf6w1ek06ms3vv3snhiwxwlDEPENDS_ON →
Information entropy quantifies uncertainty in a probability distribution via the formula H = -Σ p(x) log p(x). Remove probability theory and the distribution itself cannot be defined, making entropy inoperable. Present-tense dependency confirmed.
- cmsf6w1ek06ms3vv3snhiwxwlDERIVED_FROM →
Information entropy was derived from probability theory: Shannon entropy H = -Σ p(x) log p(x) uses probability distributions as its fundamental input. It quantifies uncertainty within a probability distribution — remove probability theory and the concept cannot be computed.
Relations to this entry
No accepted relations in this direction.
Record identity
- Created
- Aug 4, 2026, 9:00 PM UTC
- Content hash
- 0099f088942686430983d9b390f434718f34f496eadd76c150ae2905bec41424