Attention Scaling

Attention Score Normalization

Normalization of similarity scores before applying Softmax to control the probability distribution and prevent extreme attention concentrations.

← Geri