transformer

Neural networks predate transformers by decades. Transformers are a specific architecture class derived from the broader neural network research tradition. Which-came-first test: neural networks (1940s) clearly before transformers (2017).

transformer architecture contains self-attention as a core component. Per Law 10, whole-part: self-attention layers are literally parts of the transformer structure, not merely related or a kind of transformer.

The transformer architecture depends on self-attention to operate — removing self-attention means the transformer ceases to function as a transformer. Per Law 8 (DEPENDS_ON removal test): remove Y and does X stop working? Remove self-attention and the transformer has no mechanism for processing sequences. This is a present-tense operational dependency, not historical association.

The Transformer was designed to serve as the backbone architecture for language models. Per Law 8d, the question is for whose sake: the Transformer serves language model development. Self-attention within the Transformer enables modeling long-range dependencies in text — the core capability of language models.

Transformer is a specific kind of model architecture — a multi-head self-attention based neural network design. The which-kind test: a competent speaker would call a transformer a model architecture. Law 9: specific→general.

Chronological and conceptual test (Law 7): attention mechanisms existed before transformers — self-attention was introduced in 2014 (Bahdanau attention), multi-head self-attention in 2015 (Transformer paper builds on this). The transformer architecture was directly built on and fed by attention mechanisms.

Remove the attention mechanism and the transformer has no core operation — its entire architecture (self-attention layers, multi-head attention, residual connections around attention) collapses. The transformer is built around attention; without it, the architecture is just a feed-forward network.

Removal test: remove transformer architecture and kv-cache has nothing to operate on — it stops functioning entirely. kv-cache is a mechanism that exists only within the transformer inference loop.

Positional encoding was invented specifically for use in transformer architectures (Vaswani et al. 2017) — the attention mechanism lacks recurrence and thus cannot capture position on its own. Positional encoding was created to fill that gap. Law 7 which-came-first: transformer architecture predates positional encoding as a named technique.