self-attention

Neural networks existed decades before self-attention was introduced (1940s vs 2017). Self-attention is a mechanism developed within and derived from the broader neural network research program. The which-came-first test: neural networks clearly predate self-attention.

Self-attention was designed and is maintained for the sake of deep learning — its purpose is to enable parallelizable sequence processing within deep neural networks, furthering deep learning's capabilities. Per Law 8d (SERVES): the servant (self-attention) points at the master (deep learning). Its designed purpose is to further deep learning's operation.

which-came-first test (Law 7): attention mechanisms (1990s-2000s, e.g. in image processing and early NLP) predate self-attention (2014, in the transformer architecture). Self-attention is a specialized form that emerged from the broader attention mechanism concept, which fed into its design.

Self-attention is a specific kind of attention mechanism — in self-attention, the input sequence serves as both query and key, unlike cross-attention where one sequence attends to another. A competent speaker would call self-attention 'a type of attention mechanism'. Law 9: specific→general.

self-attention is a specific attention mechanism where the input and output sequences overlap — a competent ML speaker calls it 'a kind of attention mechanism'. Per Law 9: specific→general.

which-came-first-existed-and-fed-into: attention mechanism (Bahdanau 2015) predates and conceptually feeds self-attention (Vaswani et al. 2017). Self-attention is the specific case where keys, queries, and values all come from the same sequence. The general attention concept existed first and fed into the self-attention variant.

transformer architecture contains self-attention as a core component. Per Law 10, whole-part: self-attention layers are literally parts of the transformer structure, not merely related or a kind of transformer.

The transformer architecture depends on self-attention to operate — removing self-attention means the transformer ceases to function as a transformer. Per Law 8 (DEPENDS_ON removal test): remove Y and does X stop working? Remove self-attention and the transformer has no mechanism for processing sequences. This is a present-tense operational dependency, not historical association.

Positional encoding was specifically designed for self-attention architectures (like the transformer) that lack inherent sequential bias. Its entire purpose is to enable self-attention to process ordered data — it exists for the sake of self-attention's ability to handle sequences.