Positional Encoding

ALiBi (Attention with Linear Biases)

Alternative to positional encoding using linear biases directly in attention scores to represent distances between tokens.

← Geri