Attention Mechanisms Variants

Linformer Attention

Low-dimensional projection of key and value matrices to reduce complexity from O(n²) to O(n). Based on the hypothesis that attention matrices have low rank in many practical scenarios.

← Indietro