information theory

Remove probability theory — the framework of random variables, distributions, and measures — and information theory has no mathematical machinery to operate. Entropy, mutual information, and channel capacity are all defined as functions over probability distributions.

Remove mathematics — the formal language of symbols, proofs, and structures — and information theory has no formalism. Entropy formulas, coding theorems, and information measures all require mathematical structure to operate.

Mutual information was introduced by Shannon as part of information theory. It measures dependence between random variables using IT's entropy formalism. Remove information theory's framework and mutual information loses its operational machinery (Law 8). Present-tense dependency, not just historical association.

Surprisal is a measure within information theory — I(p) = -log(p). Remove IT's formalism (entropy, probability distributions) and surprisal cannot be computed or used. The removal test is positive: surprisal stops OPERATING without IT.

Rate distortion theory is a branch of information theory — it extends IT to lossy compression bounds. Remove IT's entropy and rate-distortion functions and the theory collapses. Removal test positive.

The information bottleneck method is defined through mutual information and rate-distortion trade-offs — both core information-theoretic quantities. Remove information theory and the information bottleneck ceases to operate as a measurable, operational framework.

Jensen-Shannon divergence is an information-theoretic measure — it quantifies the similarity of two probability distributions using concepts (entropy, expected log) from information theory. Remove information theory and the measure has no mathematical foundation to operate. Removal test passes.

Cross-entropy DERIVED_FROM information theory — it was derived from Shannon's entropy concept, extending the information-theoretic measure of uncertainty to the divergence between two probability distributions (Law 7). Information theory existed first and provided the foundation.

Log loss (cross-entropy loss) is derived directly from information theory. It measures the divergence between predicted probability distributions and actual outcomes using the cross-entropy formula from Shannon's information theory.

The information bottleneck principle was derived from information theory — specifically from Shannon's framework for quantifying information transfer. Claude Shannon's information theory (1948) predates and feeds into the information bottleneck method (Tishby et al., 1999). Which existed first? Information theory clearly came first.

Which-came-first: Information theory Shannon 1948 predates sparse representation concepts. The efficient coding hypothesis Barlow 1961 and information theoretic principles of sparsity directly fed into development of sparse coding Atick and Redlich 1990 and Olshausen and Field 1996. Y existed first and conceptually enabled X.

Information theory (Claude Shannon, 1948) predates and conceptually fed into the bounded-capacity framing of context windows. The idea of a bounded channel capacity directly informs how we think of a model's context window as a finite information buffer. Chronological and conceptual priority both hold.

Information theory (Shannon, 1948) existed first and conceptually fed into lossy compression. Rate-distortion theory — the mathematical foundation of lossy compression — is a direct result of information theory. Which-came-first: information theory first, lossy compression is the applied construct.

AIC was derived from information-theoretic considerations: Akaike showed that minimizing the estimated Kullback–Leibler divergence between the true model and a candidate model leads to the AIC formula. Information theory (Shannon, 1948; extended to KL divergence by Kullback and Leibler, 1951) existed before AIC (1974) and directly fed into its derivation. Historical direction: information theory came first and fed into AIC.