AI Glossary
The complete dictionary of Artificial Intelligence
Autoregressive Model
Generative model architecture that predicts the next token based on all previous tokens, building the sequence iteratively and sequentially.
Context Window
Maximum sequence size that the model can process simultaneously, limiting the amount of historical information usable for prediction.
Next Token Prediction
Fundamental objective of autoregressive models consisting of maximizing the conditional probability P(token_t|tokens_1...t-1).
Temperature Sampling
Generation technique controlling the degree of randomness in the selection of the next token by adjusting the probability distribution of logits.
Top-k Sampling
Generation method limiting selection to the k most probable tokens, avoiding low-probability tokens while maintaining diversity.
Nucleus Sampling
Dynamic selection strategy based on cumulative probability mass, adapting the number of candidates according to the model's confidence.
Beam Search
Decoding algorithm simultaneously exploring multiple candidate sequences to find the most probable global sequence.
Causal Language Model
Type of autoregressive model trained to predict future tokens based on past context, without access to future tokens during training.
Transformer Decoder-only
Neural architecture using only decoder layers with causal masking, preferred for modern autoregressive language models.
Greedy Decoding
Generation strategy systematically selecting the token with maximum probability at each step, ensuring consistency but potentially lacking creativity.
Autoregressive Generation
Text generation process where each produced token is immediately added to the context to influence the generation of subsequent tokens.
Language Model Fine-tuning
Process of specialized adaptation of a pre-trained autoregressive model on specific data to improve its performance in a targeted domain.
Zero-shot Learning
Ability of autoregressive models to accomplish tasks not seen during training by leveraging their general language knowledge.
KV Cache
Optimization mechanism storing key-value states of previous tokens to accelerate sequential autoregressive generation.
Variable Sequence Length
Ability of autoregressive models to generate sequences of different lengths dynamically adapted according to the generated content.