Document Chunking

Token-based Chunking

Segmentation strategy using tokens as the basic unit, essential for respecting the context limits of language models like GPT or BERT.

← Tillbaka