Attention Mechanisms Variants

Rotary Position Embedding (RoPE)

Positional encoding method applying matrix rotation to query and key embeddings based on their positions. Naturally integrates positional information into the attention mechanism without adding parameters.

← Back