AI Glossary
The complete dictionary of Artificial Intelligence
Neural Architecture Optimization (NAS)
Process of automating the design of optimal neural network architectures for a given task, exploring a vast search space of topologies and hyperparameters.
Architecture Search Space
Set of all possible neural network architectures, defined by constraints such as number of layers, operation types, and connection patterns.
Surrogate Model
Statistical model approximating the expensive performance function to evaluate (training a neural network) to accelerate the optimization process.
Expected Improvement (EI)
Acquisition function criterion that selects the next point to evaluate by maximizing the expectation of improvement over the current best performance.
Reinforcement Learning-based NAS
NAS approach where a controller, often a recurrent network, learns to generate neural network architectures by maximizing a performance reward.
Evolutionary NAS
NAS method inspired by biological evolution, using mutation and crossover operators on a population of architectures to find better ones.
Low-Fidelity Evaluation
Strategy for estimating architecture performance using reduced data, fewer training epochs, or a subset of the dataset to reduce costs.
Gradient-Based NAS
NAS technique that relaxes the discrete architecture selection problem into a continuous problem, allowing gradient descent to optimize architecture weights.
Hypernetwork (Hypernetwork)
A neural network whose weights are generated by another network (the hypernetwork), allowing for parameterization and optimization of a family of architectures.
Architecture Cell
A repeatable building block in a neural network architecture, whose internal structure is optimized by NAS and then stacked to form the final model.
Multi-Objective Optimization (Multi-Objective NAS)
A variant of NAS aimed at simultaneously optimizing multiple metrics, such as accuracy, latency, or energy consumption, to find optimal trade-offs.
Tree-structured Parzen Estimator Method (TPE)
A Bayesian optimization algorithm that models the distribution of good and bad configurations using Parzen tree models to guide the search.
Bandit Learning (Bandit-Based NAS)
A NAS approach treating the selection of architecture components as a multi-armed bandit problem, balancing exploration and exploitation to build the model.
Performance Proxy
A low-cost metric or model used to estimate the final performance of an architecture, avoiding a full and lengthy training phase during the search stage.
Reduced Search Space
A strategy involving limiting the architecture search space to predefined blocks or patterns to accelerate the convergence of the NAS algorithm.
Weight Sharing Between Architectures (Weight Sharing)
A technique where the weights of a neural network are shared between multiple candidate architectures being evaluated, drastically reducing the computational cost of NAS search.