CHOSEN: Compilation to Hardware Optimization Stack for Efficient Vision Transformer Inference
<|reference_start|>CHOSEN: Compilation to Hardware Optimization Stack for Efficient Vision Transformer Inference: Vision Transformers (ViTs) represent a groundbreaking shift in machine learning approaches to computer vision. Unlike traditional approaches, ViTs employ the self-attention mechanism, which has been widely used in natural language processing, to analyze image patches. Despite their advantages in modeling visual tasks, deploying ViTs on hardware platforms, notably Field-Programmable Gate Arrays (FPGAs), introduces considerable challenges. These challenges stem primarily from the non-linear calculations and high computational and memory demands of ViTs. This paper introduces CHOSEN, a software-hardware co-design framework to address these challenges and offer an automated framework for ViT deployment on the FPGAs in order to maximize performance. Our framework is built upon three fundamental contributions: multi-kernel design to maximize the bandwidth, mainly targeting benefits of multi DDR memory banks, approximate non-linear functions that exhibit minimal accuracy degradation, and efficient use of available logic blocks on the FPGA, and efficient compiler to maximize the performance and memory-efficiency of the computing kernels by presenting a novel algorithm for design space exploration to find optimal hardware configuration that achieves optimal throughput and latency. Compared to the state-of-the-art ViT accelerators, CHOSEN achieves a 1.5x and 1.42x improvement in the throughput on the DeiT-S and DeiT-B models.<|reference_end|>
@article{sadeghi2024chosen:,
    title={CHOSEN: Compilation to Hardware Optimization Stack for Efficient Vision
  Transformer Inference},
    author={Mohammad Erfan Sadeghi, Arash Fayyazi, Suhas Somashekar, Massoud
  Pedram},
    journal={arXiv preprint arXiv:2407.12736},
    year={2024},
    archivePrefix={arXiv},
    eprint={2407.12736},
    primaryClass={cs.CV cs.AI cs.AR}
}