Tokenization and Encoding

Special Token

Predefined token with a specific semantic function, such as [CLS] for classification, [SEP] for separation, or [PAD] for sequence alignment, used to structure model inputs.

← Geri