SWAP-NAS: Sample-Wise Activation Patterns for Ultra-fast NAS
<|reference_start|>SWAP-NAS: Sample-Wise Activation Patterns for Ultra-fast NAS: Training-free metrics (a.k.a. zero-cost proxies) are widely used to avoid resource-intensive neural network training, especially in Neural Architecture Search (NAS). Recent studies show that existing training-free metrics have several limitations, such as limited correlation and poor generalisation across different search spaces and tasks. Hence, we propose Sample-Wise Activation Patterns and its derivative, SWAP-Score, a novel high-performance training-free metric. It measures the expressivity of networks over a batch of input samples. The SWAP-Score is strongly correlated with ground-truth performance across various search spaces and tasks, outperforming 15 existing training-free metrics on NAS-Bench-101/201/301 and TransNAS-Bench-101. The SWAP-Score can be further enhanced by regularisation, which leads to even higher correlations in cell-based search space and enables model size control during the search. For example, Spearman's rank correlation coefficient between regularised SWAP-Score and CIFAR-100 validation accuracies on NAS-Bench-201 networks is 0.90, significantly higher than 0.80 from the second-best metric, NWOT. When integrated with an evolutionary algorithm for NAS, our SWAP-NAS achieves competitive performance on CIFAR-10 and ImageNet in approximately 6 minutes and 9 minutes of GPU time respectively.<|reference_end|>
@article{peng2024swap-nas:,
    title={SWAP-NAS: Sample-Wise Activation Patterns for Ultra-fast NAS},
    author={Yameng Peng, Andy Song, Haytham M. Fayek, Vic Ciesielski, Xiaojun
  Chang},
    journal={arXiv preprint arXiv:2403.04161},
    year={2024},
    archivePrefix={arXiv},
    eprint={2403.04161},
    primaryClass={cs.LG cs.CV cs.NE}
}