AZ-NAS: Assembling Zero-Cost Proxies for Network Architecture Search
<|reference_start|>AZ-NAS: Assembling Zero-Cost Proxies for Network Architecture Search: Training-free network architecture search (NAS) aims to discover high-performing networks with zero-cost proxies, capturing network characteristics related to the final performance. However, network rankings estimated by previous training-free NAS methods have shown weak correlations with the performance. To address this issue, we propose AZ-NAS, a novel approach that leverages the ensemble of various zero-cost proxies to enhance the correlation between a predicted ranking of networks and the ground truth substantially in terms of the performance. To achieve this, we introduce four novel zero-cost proxies that are complementary to each other, analyzing distinct traits of architectures in the views of expressivity, progressivity, trainability, and complexity. The proxy scores can be obtained simultaneously within a single forward and backward pass, making an overall NAS process highly efficient. In order to integrate the rankings predicted by our proxies effectively, we introduce a non-linear ranking aggregation method that highlights the networks highly-ranked consistently across all the proxies. Experimental results conclusively demonstrate the efficacy and efficiency of AZ-NAS, outperforming state-of-the-art methods on standard benchmarks, all while maintaining a reasonable runtime cost.<|reference_end|>
@article{lee2024az-nas:,
    title={AZ-NAS: Assembling Zero-Cost Proxies for Network Architecture Search},
    author={Junghyup Lee, Bumsub Ham},
    journal={arXiv preprint arXiv:2403.19232},
    year={2024},
    archivePrefix={arXiv},
    eprint={2403.19232},
    primaryClass={cs.CV cs.LG}
}