A feed-forward network (FFN) is a neural network module consisting of two linear projections separated by a non-linear activation function (ReLU or GELU). Given an input vector x and hidden dimension d_hidden, it computes FFN(x) = W2·activation(W1·x + b1) + b2. It is applied independently and identically to each position in a sequence. In transformer architectures, it follows the self-attention layer and provides the non-linear feature transformation that enables the model to learn complex representations beyond what linear attention alone can express. Its parameters are: input dimension d_model, hidden dimension d_hidden (typically 4×d_model), and the activation function. It persists through training as learnable weights and is deployed in every transformer-based model.\n\n[formal: feed-forward network | substrate: behavior | horizon: hours | explicit: yes | epoch: 0.01]
Accepted ontology entry
feed-forward network
A feed-forward network (FFN) is a neural network module consisting of two linear projections separated by a non-linear activation function (ReLU or GELU). Given an input vector x and hidden dimension d_hidden, it computes FFN(x) = W2·activ…
Definition
Why it is in scope
A human-made neural network module that maps each input vector to an output through two linear transformations with a non-linear activation between them. Used as the per-position transformation component in transformer architectures, it extends representational capacity beyond linear attention by introducing pointwise non-linearity.
Names and aliases
- feed-forward networken · CANONICAL
Relations from this entry
- cmrg0pc1x00e82a1nppcnwovjINSTANCE_OF →
A feed-forward network is a specific kind of neural network architecture where information flows in one direction from input to output without cycles. A competent speaker would call it 'a neural network' — it's a specific instance of the general architecture pattern (Law 9).
- cmsfvi9eg00l3qszgv2mfg1u9DEPENDS_ON →
A feed-forward network requires activation functions to operate — without them, neurons collapse to linear transforms and the network cannot model non-linear relationships. The which-came-first test passes: activation function (epoch 0.79) predates modern feed-forward networks (epoch 0.85). This is constitutive at the object level: remove activation functions and the feed-forward network ceases to function as intended.
Relations to this entry
No accepted relations in this direction.
Record identity
- Created
- Aug 10, 2026, 1:12 PM UTC
- Content hash
- 7d3cd4ef865b374a6613a19a54950d8073cba4144d4c24ade7db99716d67bbc9