Attention Mechanism

Executive Definition: Attention Mechanism — A technique in neural networks that allows models to dynamically focus on specific parts of the input sequence.

The attention mechanism calculates relevance scores between different tokens in data, allowing the model to focus on the most important context when generating an output. In natural language processing, it helps resolve long-range dependencies and is the foundational concept behind the Transformer architecture.

Frequently Asked Questions

What is self-attention?

Self-attention evaluates the relationships between all elements within a single sequence. It allows the model to compute a representation of a sequence by relating different positions to one another.

Why was the attention mechanism a breakthrough?

Prior architectures like RNNs struggled with long-range dependencies because data degraded over sequential steps. Attention calculates direct connections between all tokens, solving the bottleneck.

What is multi-head attention?

Multi-head attention applies several attention mechanisms in parallel. This allows the model to simultaneously process different aspects of the input, such as syntax and semantics, independently.