entropy

Entropy operates on probability distributions — remove probability theory as a concept and entropy loses its mathematical framework entirely. The removal test passes: entropy cannot function without the concept of probability.

Entropy as a concept (especially Shannon entropy in information theory) needs the concept of information to operate: entropy measures uncertainty/information content. Remove information and entropy as an informational metric ceases to function. This is a present-tense operational dependency, not merely historical or sayable.

Spectral entropy is Shannon entropy applied to power spectral density. It was derived by taking the entropy concept from information theory and applying it to the frequency domain representation of signals. Entropy as a concept predates and feeds into spectral entropy.

Rényi entropy (1961) was introduced as a one-parameter generalization of Shannon entropy (1948). Shannon entropy is the α→1 limit of the Rényi formula. Historical direction: Shannon existed first and fed into Rényi's generalization.

Cross-entropy H(p,q) = -Σ p(x) log q(x) is the extension of Shannon entropy H(p) = -Σ p(x) log p(x) to two distributions. The entropy concept (Shannon 1948) existed first and cross-entropy was derived from it — the cross-entropy formula is literally the negative expected log of q, while entropy is the same form with p=q. Entropy is the more fundamental construct that cross-entropy derives from.

KL-divergence D_KL(P||Q) = Σ P(x) log(P(x)/Q(x)) = -Σ P(x) log Q(x) + Σ P(x) log P(x) = cross-entropy(P,Q) - entropy(P). The KL formula is literally decomposed into Shannon entropy terms. It was derived from the entropy concept by Kullback and Leibler (1951). Entropy (Shannon 1948) predates and feeds into KL-divergence.

Conditional entropy H(Y|X) extends Shannon entropy H(X) to the setting of dependent variables. Entropy existed first (Shannon 1948) and conditional entropy was derived from it by introducing the conditioning variable. The chain rule H(X,Y)=H(X)+H(Y|X) shows conditional entropy as the extension of entropy to joint distributions.

Conditional entropy is a specific kind of entropy — the entropy of one random variable given knowledge of another. A competent speaker would call it 'an entropy measure' — it is a specific case within the general concept of entropy.

Differential entropy is the specific instance of Shannon entropy applied to continuous (density-based) distributions rather than discrete ones, nearest kind being entropy itself.

Differential entropy is the continuous analog of Shannon (discrete) entropy, replacing summation with integration over a probability density. It was derived from Shannon entropy to handle continuous distributions — entropy came first and gives rise to differential entropy as its continuous extension.