  Since introduced, Swin Transformer has achieved remarkable results in the
field of computer vision, it has sparked the need for dedicated hardware
accelerators, specifically catering to edge computing demands. For the
advantages of flexibility, low power consumption, FPGAs have been widely
employed to accelerate the inference of convolutional neural networks (CNNs)
and show potential in Transformer-based models. Unlike CNNs, which mainly
involve multiply and accumulate (MAC) operations, Transformer involve
non-linear computations such as Layer Normalization (LN), Softmax, and GELU.
These nonlinear computations do pose challenges for accelerator design. In this
paper, to propose an efficient FPGA-based hardware accelerator for Swin
Transformer, we focused on using different strategies to deal with these
nonlinear calculations and efficiently handling MAC computations to achieve the
best acceleration results. We replaced LN with BN, Given that Batch
Normalization (BN) can be fused with linear layers during inference to optimize
inference efficiency. The modified Swin-T, Swin-S, and Swin-B respectively
achieved Top-1 accuracy rates of 80.7%, 82.7%, and 82.8% in ImageNet.
Furthermore, We employed strategies for approximate computation to design
hardware-friendly architectures for Softmax and GELU computations. We also
designed an efficient Matrix Multiplication Unit to handle all linear
computations in Swin Transformer. As a conclude, compared with CPU (AMD Ryzen
5700X), our accelerator achieved 1.76x, 1.66x, and 1.25x speedup and achieved
20.45x, 18.60x, and 14.63x energy efficiency (FPS/power consumption)
improvement on Swin-T, Swin-S, and Swin-B models, respectively. Compared to GPU
(Nvidia RTX 2080 Ti), we achieved 5.05x, 4.42x, and 3.00x energy efficiency
improvement respectively. As far as we know, the accelerator we proposed is the
fastest FPGA-based accelerator for Swin Transformer.

Hardware Architecture