Neural Architecture Search: A Survey – Overview, Methods, and Future Directions

Abstract: Neural Architecture Search (NAS) aims to automate the design of neural network architectures, a process traditionally performed by human experts. This survey classifies existing NAS work along three dimensions—search space, search strategy, and performance evaluation—and discusses their evolution.

Introduction: The rapid progress of deep learning in perception tasks is largely due to hand‑crafted architectures, which are costly to design. NAS treats architecture design as an AutoML problem, overlapping with hyper‑parameter optimization and meta‑learning.

Search Space: Search spaces define the set of possible network structures. They range from simple chain‑structured networks to complex multi‑branch and cell‑based designs, including hierarchical and layered spaces. Recent work explores both bounded and unbounded spaces, conditional parameterizations, and the trade‑off between expressiveness and bias.

Search Strategies: Strategies determine how to explore the search space. Approaches include grid/random search, evolutionary algorithms, Bayesian optimization, reinforcement learning, gradient‑based methods, and one‑shot models. Historical methods and recent improvements that reduce computational cost are described.

Performance Evaluation: Evaluating candidate architectures typically requires full training, which is expensive. Low‑fidelity proxies (short training, reduced data, smaller models), learning‑curve extrapolation, weight‑sharing, and surrogate models are surveyed as ways to accelerate evaluation.

Future Research Directions: Promising avenues include extending NAS beyond image classification to language modeling, reinforcement learning, and multimodal tasks; developing multi‑task and multi‑objective NAS; designing more flexible and general search spaces; establishing common benchmarks; integrating NAS into full AutoML pipelines; and gaining deeper insight into why certain architectures succeed.