Neural Architecture Search (NAS) treats network design as an optimization problem: try candidates, train them, keep the best, and start over for the next task. Neural Architecture Generation (NAG) instead trains a generative model that has learned what good architectures look like, and samples new ones on demand. The sections below explain the difference, with small experiments that run in your browser. Open any section to read it.
Many of deep learning's landmark results came from a new architecture rather than more data or a new optimizer.
Deep convolutional networks won ImageNet (Krizhevsky et al., 2017), residual connections made very deep networks trainable (He et al., 2016), and attention replaced recurrence in sequence models (Vaswani et al., 2017). Each of these was designed by hand, by choosing layer types, their order, how they connect, and many hyperparameters.
The design space these choices define is enormous and highly structured, and exploring it by trial and error is slow (Bergstra et al., 2013). As models grew, manual design became a bottleneck for scale, reproducibility and reuse across tasks. The natural next step was to let an algorithm do the exploring.
That algorithm has taken two broad forms. The first, search, looks for the single best architecture for one task. The second, generation, learns a model of good architectures that can be reused.
Neural Architecture Search frames design as a discrete optimization problem over a space of candidates that a human defined in advance (Elsken et al., 2019).
Written out, NAS solves a bi-level problem. The outer level looks for the architecture \(A\) in the search space \(\mathcal{S}\) with the highest reward \(R\). The inner level trains the weights \(w\) of each candidate:
\[ A^* = \arg\max_{A \in \mathcal{S}} R\big(A, w^*(A)\big), \qquad w^*(A) = \arg\min_{w} \mathcal{L}_{\text{train}}(w, A). \]Every evaluation of \(R\) hides a full training run, which is why early NAS cost thousands of GPU-days (Real et al., 2017). Weight sharing (Pham et al., 2018), low-fidelity training (Li et al., 2018) and learning-curve extrapolation (Domhan et al., 2015) made each evaluation cheaper, but evaluation still plays the same role: an external score that decides which candidates survive.
Every demo on this page uses the same small cell, in the style of NAS-Bench-201 (Dong & Yang, 2020). It has four nodes, and each of its six edges carries one of five operations: none, skip connection, 1×1 convolution, 3×3 convolution or average pooling. That makes 56 = 15,625 cells, few enough for the browser to score every one of them. That is how the demos can show exact answers.
Accuracy of all 15,625 cells on the selected task. The marker shows the cell on the left.
The accuracies are a hand-designed synthetic function, not NAS-Bench-201 data. It rewards convolutions, depth with a parallel skip connection, and extra capacity on harder tasks, and it adds a small deterministic "noise". This page uses it to illustrate ideas, not to rank methods.
As the field matured, the costs of this framing became clearer (White et al., 2023):
These limits have led some authors to ask whether NAS is full automation or optimization inside human-imposed boundaries (Baymurzina et al., 2022).
Neural Architecture Generation treats architectures as samples from a learned probability distribution \(p_\theta(A)\), and trains a generative model to put most of its probability on good designs.
Instead of an \(\arg\max\), the target becomes a distribution that favors high reward. A convenient choice weighs every architecture exponentially by its reward (Soin et al., 2025):
\[ p^*(A) \propto \exp\big(\beta\, R(A)\big). \]The temperature parameter \(\beta \ge 0\) controls how picky the distribution is. With \(\beta = 0\) every architecture is equally likely. As \(\beta \to \infty\), all the mass collapses onto the single best architecture, and sampling from \(p^*\) gives back exactly the NAS answer \(A^*\). In this sense, NAS is the zero-temperature limit of NAG.
A generator is then trained to approximate this target, \(\min_\theta D\big(p_\theta \,\|\, p^*\big)\) for some divergence \(D\). Once trained, it can be conditioned on a description of the dataset \(D\) or on constraints \(C\) such as a latency budget, and draw architectures from \(p_\theta(A \mid D, C)\) (Lee et al., 2021) (Lomurno et al., 2024).
This demo computes the exact \(p^*(A)\) over all 15,625 cells as you move the slider.
Probability mass of \(p^*\) in each accuracy bin (0.5-point bins). The dashed marker is the best cell, \(A^*\).
| Expected accuracy | |
|---|---|
| Probability of drawing the best cell | |
| Effective number of architectures (exp of the entropy) |
Keeping a distribution rather than a point is not only more general. It makes diversity, uncertainty and trade-offs visible and controllable, and it is what allows the learned model to be reused, as the next section shows.
A search pays its full cost on every new task. A generator pays a large cost once, during training, and afterwards produces architectures for new tasks by inference alone.
This is amortized inference. The expensive exploration of architecture–performance pairs is done once, across many training tasks, and stored in the weights of a shared model \(p_\theta(A \mid D)\). Given the descriptor of a new dataset, the model proposes architectures in a forward pass (Lee et al., 2021) (Hemmi et al., 2024). The same idea underlies "train once, specialize many times" approaches to deployment (Cai et al., 2020).
The toy tasks are described by two numbers, difficulty and input resolution, and the best architecture changes with them. When this section opens, a small conditional generator is trained on 16 training tasks, using the evaluations of one 300-step evolutionary search per task, weighted as in \(p^* \propto \exp(\beta R)\). Then pick any new task and compare.
The generator trains when this section opens.
A NAG method can be read along three interdependent axes: how architectures are represented, which mechanism produces them, and what guides the generator toward good ones.
Architectures are graphs, usually directed acyclic graphs of operations. A generator needs a space in which to model them. The options include:
Unlike a search space, which only lists what is allowed, a representation space models how architectures relate to each other. Whether such representations help even without supervision is a question of its own (Yan et al., 2020).
Almost every family of generative models has been applied to architectures. The four mini-demos below illustrate the main ideas on the toy space. Each is an illustration of the principle, not a reimplementation of the cited methods.
Methods such as NAO (Luo et al., 2019) embed architectures in a continuous space, train a predictor of accuracy on it, climb the predictor's gradient, and decode the result. Here, the 2-D map's horizontal axis was fitted to predict accuracy and its vertical axis to predict size, from 120 evaluated cells. Each pixel is colored by the true accuracy of the cell it decodes to.
What to look for: many pixels decode to the same cell, and whole regions decode to degenerate cells. That is the mismatch between a continuous space and discrete graphs. The climb also stops where the smooth predictor peaks, which is not where the true best cell is.
Diffusion models learn to reverse a gradual noising process (Ho et al., 2020). Applied to architecture graphs, a predictor can steer the denoising toward high accuracy (An et al., 2024). Here, the training data is an archive of 300 good cells, noise replaces an edge's operation at random, and a linear predictor fitted on the archive provides the guidance.
Gray bars: accuracy of the 300 archive cells. After "Generate 300", orange bars: the generated cells. The markers show the archive's best cell and the true best cell.
| Mean accuracy / best | – |
|---|---|
| Validity | – |
| Uniqueness | – |
| Novelty (not in the archive) | – |
What to look for: the exact denoiser is the best possible fit to its training data, yet it can only reproduce archive cells (novelty 0) (Carlini et al., 2023). The per-edge denoiser recombines edges into new cells, and with guidance it can reach cells better than anything in the archive. Stronger guidance raises accuracy and lowers uniqueness.
A Generative Flow Network builds an object step by step and is trained so that complete objects are sampled with probability proportional to their reward (Bengio et al., 2026), which has been applied to architecture generation (Soin et al., 2025). Here, the same policy builds a cell one edge at a time, trained with two objectives: reinforcement learning, which maximizes expected reward, and a GFlowNet objective (trajectory balance) with reward \(R = \exp(\beta\,\text{accuracy})\).
Press "Train both". Training runs 200,000 construction episodes per policy.
| GFlowNet | Reinforcement learning | |
|---|---|---|
| Expected accuracy | – | – |
| Effective number of architectures | – | – |
| Distinct top-1% cells in 1,000 samples | – | – |
What to look for: reinforcement learning converges on a single cell, a point estimate, just like search. The GFlowNet keeps probability on many good cells, roughly in proportion to their reward, so its samples stay diverse.
Deployment adds constraints such as size, latency or energy. Instead of one search per budget, Pareto-aware generators learn \(p_\theta(A \mid \lambda)\) for a whole range of budgets \(\lambda\) (Guo et al., 2022) (Lomurno et al., 2024). Here, a generator was trained on ten budgets. Move the slider to ask for any budget, including ones it never saw.
What to look for: the samples follow the budget as it moves, with no retraining and no search. Not every sample fits, because this small generator picks each edge independently, but a cell's size is free to compute, so over-budget samples are simply discarded before any training.
Other generator families include adversarial generators (Rezaei et al., 2021), graph networks that emit a whole architecture in one step (Agiollo & Omicini, 2022) (Liang & Sun, 2022), diffusion variants with evolutionary (Zhou et al., 2025) or progressive (Zhang & Zhuang, 2025) sampling, and language models fine-tuned on architecture codes (Poddenige et al., 2025).
What pulls a generator toward good architectures? The studies fall into two groups:
Either way, the defining move from NAS to NAG is that the performance signal is internalized into the generator's parameters instead of acting as an external filter after the fact.
Some systems proposed and critiqued by large language models (Rahman & Chakraborty, 2024) (Yang et al., 2025) escape fixed search spaces, but still rely on iterative evaluate-and-refine loops, so they behave more like search than like a learned generative model.
NAS results are usually summarized by the best architecture found and its cost. A generator produces a distribution, and a single number cannot describe a distribution.
Tabular benchmarks such as NAS-Bench-101 (Ying et al., 2019) and NAS-Bench-201 (Dong & Yang, 2020) store the trained accuracy of every architecture in a fixed space, so any method can be scored by lookup. That gives reproducibility and fair comparison, but only if the generator is confined to that fixed space, which removes the thing NAG is supposed to be good at: generating beyond it.
The alternatives have their own problems. Training every generated model makes results depend on training recipes (Yang et al., 2020). Scoring with a surrogate predictor imports the predictor's biases, and a predictor trained on familiar designs tends to undervalue novel ones, the same flaw found in feature-based metrics for images (Stein et al., 2023).
NAG is young, and several challenges still limit its reliability and adoption.
Continuous latent spaces and stochastic denoising can produce graphs that are not valid networks, so samples often need repair or rejection. Building constraints such as acyclicity and connectivity into the generation process itself is an open direction. Discrete learned tokens are one early attempt (Poddenige et al., 2025).
Most generators still operate over the fixed operation sets of benchmark spaces, so gains may reflect better interpolation within a table rather than new design principles. Evaluation protocols built for generative models, measuring diversity, transfer and controllability, are still missing.
Architecture–performance landscapes are large and rugged, and a single surrogate predictor rarely generalizes across them, so generators can end up optimizing the predictor's errors. Progressive guidance from ensembles of weaker predictors is one response (Zhang & Zhuang, 2025).
Ideally, one generator would map any hardware budget to a near-optimal architecture and share structure across budgets, as in the budget demo, instead of treating each budget as a separate problem (Guo et al., 2022) (Lomurno et al., 2024).
Most generators need thousands of evaluated architectures before they are useful. Priors extracted from existing benchmarks (Xie et al., 2025) and training-light evolutionary dynamics (Zhou et al., 2025) reduce this cold start. The long-term goal is a foundation distribution over architectures that works across tasks.
Generated architectures rarely come with a reason. Agent-style critique loops that justify each design choice (Yang et al., 2025) hint at generators whose decisions can be inspected, which matters for trust and for learning something from what they produce.
This page is based on From Search to Synthesis: A Systematic Mapping Study of Neural Architecture Generation, by Thiago C. S. Barbieri, Edesio Alcobaça and André C. P. L. F. de Carvalho (Institute of Mathematical and Computer Sciences, University of São Paulo). The manuscript is currently under review, and a link will be added here once it is published.
The page draws on the study's framing and bibliography. The demos and all their numbers were made for this page.