KI-Glossar
Das vollständige Wörterbuch der Künstlichen Intelligenz
Sinusoidal Positional Encoding
Positional encoding method using sinusoidal functions of different frequencies to create unique and deterministic position representations without parameter learning.
Learned Positional Encoding
Approach where position embeddings are learned as trainable model parameters, allowing adaptive optimization to specific training data.
Relative Positional Encoding
Advanced technique that encodes relative distances between tokens rather than their absolute positions, improving generalization to variable sequence lengths.
Absolute Positional Encoding
Traditional positional encoding method where each position in the sequence receives a unique embedding based on its absolute index in the sequence.
Rotary Positional Encoding (RoPE)
Innovative technique that applies matrix rotation to query and key embeddings, effectively integrating position information directly into the attention mechanism.
Alibi Positional Encoding
Method that penalizes attention scores based on the distance between tokens, enabling effective extrapolation to longer sequence lengths without retraining.
Position Embeddings
Dense vectors representing the position of each token in a sequence, added or concatenated to token embeddings to provide spatial or temporal location information.
Attention with Positional Encoding
Integration of positional encoding into the attention mechanism to allow the model to weight tokens differently based on their relative positions in the sequence.
BERT Positional Embeddings
Specific implementation of learned positional encoding in the BERT architecture, using trainable position embeddings with a fixed maximum sequence length of 512 tokens.
GPT Positional Encoding
Positional encoding system used in GPT models, initially based on learned position embeddings to effectively model directional dependencies in text.
Transformer Positional Encoding
Essential component of the original Transformer architecture using sinusoidal encodings to allow the model to use token order without recurrent mechanisms.
3D Positional Encoding
Extension of positional encoding to three-dimensional data like volumes or videos, incorporating position information on three spatial or temporal axes.
Complex Positional Encoding
Advanced variant using complex numbers to represent positions, enabling richer modeling of spatial relationships and multiple frequencies.
Hierarchical Positional Encoding
Structured approach that encodes positions at multiple levels of granularity, capturing both local and global positions in the sequence.
T5 Positional Encoding
Specific implementation in the T5 architecture using scalar position embeddings added to token embeddings, designed to simplify the architecture while maintaining performance.
XLNet Relative Positional Encoding
Sophisticated mechanism in XLNet that models relative distances between tokens in attention computation, enabling better generalization across different sequence lengths.
DeBERTa Disentangled Attention
Innovation in DeBERTa that explicitly separates content and position in the attention mechanism, using disentangled positional encoding to improve representation.
Longformer Positional Encoding
Positional encoding system adapted for processing long sequences, combining global and local position embeddings to optimize sliding window attention.
Reformer Locality Sensitive Hashing
Specialized technique in Reformer that uses LSH with positional encoding to reduce computational complexity of attention on very long sequences.