Natural gradient is a human-made optimization algorithm that modifies standard gradient descent by multiplying the gradient by the inverse Fisher information matrix, effectively following the steepest descent in the Riemannian manifold of probability distributions. Parameters: (1) loss function L(θ) defining the objective, (2) Fisher information matrix G(θ) derived from the model's predictive distribution, (3) learning rate η > 0, (4) optional damping for numerical stability. The update rule is θ_{t+1} = θ_t - η G(θ_t)^{-1} ∇L(θ_t). It persists as a documented algorithm in machine learning literature, implemented in optimization libraries, and used for training neural networks and variational inference where parameter geometry matters. [formal: natural_gradient | substrate: mind | horizon: generations | explicit: yes | epoch: 0.02]
Accepted ontology entry
natural-gradient
Natural gradient is a human-made optimization algorithm that modifies standard gradient descent by multiplying the gradient by the inverse Fisher information matrix, effectively following the steepest descent in the Riemannian manifold of…
Definition
Why it is in scope
A human-made optimization method in machine learning that preconditions gradient descent using the Fisher information metric, persisting as a standard technique for training models with parameter-space geometry.
Names and aliases
- natural-gradienten · CANONICAL
Relations from this entry
- fisher-informationDEPENDS_ON →
Natural gradient computes the update θ_{t+1}=θ_t - η G(θ)^{-1}∇L(θ) where G(θ) is the Fisher information matrix. Remove Fisher information and the algorithm has no metric to precondition the gradient; it ceases to operate as natural gradient, reverting to ordinary gradient descent. Operational removal test per Law 8b.
- cmsi4l2p80452ywh500rxvdvfSERVES →
Natural gradient is an optimization algorithm that uses the Fisher information metric to precondition gradients. It is built and maintained for the sake of optimization — specifically for more efficient optimization in curved parameter spaces. Law 8d: servant (natural gradient) points at master (optimization).
- cmsllb31s07h4nobp1h0i7lu3INSTANCE_OF →
Natural gradient is a specific optimization algorithm that minimizes loss functions by preconditioning gradients with the Fisher information metric. Law 9: specific→nearest kind. A competent speaker calls natural gradient 'an optimization algorithm'. Nearest kind is optimization algorithm, not a more general category like algorithm.
- cmsd5x8jv03ih3vv3xpnf9qdkDERIVED_FROM →
Natural gradient was developed as a generalization of gradient descent that respects the information geometry of the parameter space. Which-came-first test per Law 7: gradient descent (mid-20th century) predates Amari's natural gradient (1970s). Natural gradient inherits the iterative descent structure of gradient descent and replaces the Euclidean metric with the Fisher information metric.
Relations to this entry
No accepted relations in this direction.
Record identity
- Created
- Sep 5, 2026, 12:13 AM UTC
- Content hash
- aa4b61deadacb6a53df1160b7444581e6095ce49be4f81dda176fe64ccec1494