AI Glossary
The complete dictionary of Artificial Intelligence
Token Alignment
Process by which cross-attention learns to automatically align significant tokens or segments between two sequences of different lengths or structures. Crucial for translation tasks where correspondences are not bijective.
Sparse Cross-Attention
Optimization of cross-attention limiting attentional connections to predefined or learned subsets of relevant positions. Reduces computational complexity from O(n²) to O(n log n) or O(n) for long sequences.
Cross-Attention with Relative Position
Extension of cross-attention incorporating relative position information between elements of the two sequences rather than absolute positions. Improves generalization to sequence lengths not seen during training.
Adaptive Cross-Attention
Attention mechanism dynamically adjusting its focus based on context or the model's internal state. Enables flexible allocation of attentional resources according to the complexity or importance of inter-sequence regions.
Cross-Attention Pooling
Aggregation technique using cross-attention to selectively weight and combine features from a target sequence based on a query sequence. Generates globally informed contextual representations for classification or regression.
Bilateral Cross-Attention
Symmetric architecture applying cross-attention in both directions between two sequences, enabling complete bidirectional interaction. Used in tasks requiring mutual alignment such as paraphrasing or semantic matching.
Cross-Attention Regularization
Constraint techniques applied to cross-attention weights to encourage desirable properties such as sparsity, diversity, or temporal coherence. Improves model interpretability and generalization.
Memory-Augmented Cross-Attention
Extension of cross-attention integrating external or persistent memory accessible via attention mechanisms. Allows storing and retrieving information beyond the immediate context window for long-range tasks.