## Designing Efficient LLM Accelerators for Edge Devices

Jude Haris, Rappy Saha, Wenhao Hu, José Cano School of Computing Science, University of Glasgow, Scotland, UK

Abstract -The increase in open-source availability of Large Language Models (LLMs) has enabled users to deploy them on more and more resource-constrained edge devices to reduce reliance on network connections and provide more privacy. However, the high computation and memory demands of LLMs make their execution on resource-constrained edge devices challenging and inefficient. To address this issue, designing new and efficient edge accelerators for LLM inference is crucial. FPGA-based accelerators are ideal for LLM acceleration due to their reconfigurability, as they enable model-specific optimizations and higher performance per watt. However, creating and integrating FPGAbased accelerators for LLMs (particularly on edge devices) has proven challenging, mainly due to the limited hardware design flows for LLMs in existing FPGA platforms.

specialized towards running LLMs on edge devices, supporting LLM inference on commodity CPUs and GPUs.

To tackle this issue, in this paper we first propose a new design platform, named SECDA-LLM , that utilizes the SECDA methodology to streamline the process of designing, integrating, and deploying efficient FPGA-based LLM accelerators for the llama.cpp inference framework. We then demonstrate, through a case study, the potential benefits of SECDA-LLM by creating a new MatMul accelerator that supports block floating point quantized operations for LLMs. Our initial accelerator design, deployed on the PYNQ-Z1 board, reduces latency ( 1 7 . seconds per token or ∼ 2 seconds per word) by 11 × over the dual-core Arm NEON-based CPU execution for the TinyLlama model.

## I. INTRODUCTION

Large language models (LLMs) are an emerging class of machine learning (ML) systems geared toward learning from huge text-based datasets. LLMs such as GPT-3 [1] have revolutionized the ability of Artificial Integillence (AI) systems to understand and generate human language. Due to innovative changes in model architecture, training methods, and through the help of the popularity of online services like ChatGPT [2], the field of LLMs is evolving rapidly.

The number of everyday users is also growing rapidly due to the myriad of use cases from translation [3], classification [4], code generation [5] to healthcare [6]. Additionally, cloudbased LLM services are currently the go-to method of access to LLMs for everyday users, but as the availability of opensource LLMs and datasets increases, especially over the last few years, the need for edge-based, localized access and execution of LLMs has become more sought after. Massive community-driven pushes have facilitated easy access to LLMs and rapid prototyping of new models and optimizations to enable efficient LLM inference on edge devices. At the forefront of these pushes is the GPT-Generated Model Language [7] (GGML). GGML is a tensor library for ML specialized in enabling large models and high performance on commodity hardware. Furthermore, the GGML 's llama.cpp project [8] is

Unfortunately, LLMs can be very computationally demanding, even for inference. In addition, due to their large memory footprint, they require high memory capacity and bandwidth. These properties of LLMs make them challenging to execute on resource-constrained edge devices. For example, running LLMs on mobile phones or Internet-of-Things devices (IoT) devices is in some cases impossible due to memory constraints. Hence, there is a great demand for developing and deploying custom hardware accelerators to efficiently run these LLMs on resource-constrained edge devices. Fortunately, Field Programmable Gate Arrays (FPGAs) are ideal for designing new flexible and power-efficient accelerators that can take advantage of LLM optimizations, such as block floating point quantization. While some FPGA-based accelerators [9], [10] already exist for LLM inference at the edge, with constant changes to LLM architectures and optimizations we are in need of new specialized FPGA-based accelerators.

To create new and innovative FPGA-based accelerator architectures for LLM inference at the edge, we need ways to quickly prototype and evaluate LLM-based inference accelerators to reduce development costs and increase design space exploration. Hence, we propose SECDA-LLM, a new platform for designing, integrating and deploying specialized accelerators for LLMs at the edge. SECDA-LLM employs the SECDA design methodology [11], and similar to SECDA-TFLite [12], it provides the user with the ability to quickly prototype accelerator designs with the target application framework, in this case, llama.cpp project. Our SECDA-LLM platform enables the designer to consider hardware-software co-design optimizations in terms of both algorithmic and hardware implementations [13], and makes deployment of LLMs through FPGA-based accelerators effortless. The contributions of this work are as follows:

- · SECDA-LLM , a new design platform using the SECDA methodology which enables the design, integration and deployment of FPGA-based accelerators for LLMs on resource-constrained edge devices.
- · A case study to demonstrate SECDA-LLM , where we prototype and deploy a new accelerator to efficiently execute quantized MatMul operations during LLM inference.
- · Evaluation of our initial accelerator design executing the TinyLlama model [14] on the PYNQ-Z1 [15] board, where we achieve a 11 × speedup over dual-core ARM NEON-based CPU execution.

## II. BACKGROUND AND RELATED WORK

## A. Large Language Models

LLMs are a family of ML models that use the Transformer [16] architecture as the key component, and are pretrained on large amounts of language data. People usually use them by fine-tuning with downstream task-specific datasets. LLMs usually have large amounts of parameters. For example, Llama is designed to start with 7B parameters [17]. Also, many types of language-generating LLMs auto-regressively compute the next tokens (chunks of text) by using the previously cached information. This computation paradigm, called KV cache [18], introduces a large amount of memory overheads while improving performance.

Quantization techniques are extensively employed to deploy parameter-intensive LLMs on resource-constrained edge devices. For example, utilizing 8-bit quantization facilitates the retention of model accuracy while significantly reducing the model size [19], [20]. In addition, prior studies have demonstrated efforts in experimenting with smaller number of bits, such as 4-bit quantization, to enable the operation of LLMs on edge devices while upholding accuracy levels [21]. Another technique that can be considered is block floating point (BFP) quantization, and there have been some works [22] comparing the efficacy of BPF to traditional integer (8/4-bit) quantization.

## B. Inference framework: llama.cpp

llama.cpp [8] is a pure C/C++ library with minimal external dependencies for enabling LLMs inference on a wide range of hardware. Currently, llama.cpp supports a wide range of LLMs, including some multi-modal and custom-defined models. Additionally, it supports 1.5-bit, 2-bit, 3-bit, 4-bit, 5-bit, 6-bit, and 8-bit BFP quantization. In llama.cpp , BFP quantization is leveraged to quantize the weights of the LLMs, and it includes few quantization variations. These variations are typically denoted as Qx y \_ , where x represents the number of bits per weight and y denotes the type of quantization.

llama.cpp [8] employs the GPT-Generated Unified Format model format (GGUF) to represent LLMs. Within this format, it is possible to represent the weights of an LLM with as few as 1.5 bits using BFP quantization. These quantized weights enable users to run LLMs on resource-constrained edge devices such as the Raspberry Pi and the Pixel phone [23].

LLM inference is possible on edge devices equipped with CPUs or GPUs due to the optimized support provided by llama.cpp [8] for A VX, AVX2, and AVX-512 on x86 architectures as well as custom CUDA kernels for running on NVIDIA GPUs. However, LLM inference on resource-constrained devices, especially on FPGAs is not straightforward, as the design process for new FPGA-based accelerators has not been integrated with inference platforms like llama.cpp yet.

## C. FPGA-based acceleration for LLMs

Previous works have described acceleration ideas for running LLMs on FPGAs [24]. The focus was generally to accelerate LLM inference. Additionally, they have addressed the

Fig. 1: Overview of the SECDA methodology [11]. Components in the dashed lines correspond to simulation, and in the dotted lines to execution on real hardware.

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00462v1-figure-ac69ecfe-e080-4076-8687-52f1edd73645.jpg)

challenge of executing a wide variation of LLMs on FPGAs by proposing an overlay FPGA-based processor [9]. However, research has yet to focus on developing design flows and enabling explorations of new FPGA-based acceleration ideas for the latest LLMs. While transformer accelerator design that commences with the C/C++ code of an LLM exists [10], their primary focus lies on the accelerator design aspect, not necessarily emphasizing the design process itself, providing no design methodology (e.g., no hardware simulation for prototyping) for accelerator development.

As such, we aimed to employ SECDA (SystemC Enabled Co-design of DNN Accelerators) 1, a hardware-software design methodology to efficiently produce optimized inference accelerators for edge devices using FPGAs. SECDA uses SystemC [25] as an accelerator simulation framework, allowing candidate designs to be efficiently iterated upon. Additionally, SECDA uses SystemC High-Level Synthesis (HLS) to produce a synthesizable design based on the same SystemC accelerator definition. One key aspect of SECDA is the full integration of the design process with the target application framework. For LLMs inference, llama.cpp is the ideal target application framework. With the integration of llama.cpp as the application framework, it becomes feasible to convert an LLM to the GGUF format and execute LLMs on edge FPGAbased platforms.

## III. SECDA-LLM

SECDA-LLM is a specialized platform for creating FPGAbased LLM accelerators for edge devices using the SECDA methodology [11] within the llama.cpp environment. Figure 2 outlines the main components of SECDA-LLM. The platform simplifies the accelerator design process by integrating the SECDA tools (e.g., AXI-API, profiler) , thus allowing a seamless connection between the SECDA design environment and the target application framework, llama.cpp . This integration enables developers to begin prototyping and integrating their new accelerator designs with minimal setup costs.

The rest of this section provides details about SECDA-LLM at: i) how it is integrated with llama.cpp ; ii) how it enables the accelerator designer to prototype and simulate new designs using SystemC [25]; iii) the ease of hardware evaluation; iv) the profiling and performance analysis capabilities.

Fig. 2: Overview of SECDA-LLM. Key SECDA components are highlighted in orange, and the LLM components are highlighted in beige.

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00462v1-figure-22452794-c68d-4cdb-9281-6c849e3cb144.jpg)

## A. Integration with llama.cpp

Figure 2 shows that SECDA-LLM builds upon the core llama.cpp project. Our current integration is through llama.cpp 's 'main' example project, which enables users to run LLMs with minimal overhead. We can connect into llama.cpp once it calls any of the GGML 's operations.

Depending on our target operation(s), we create additional connection points from the GGML library to the SECDA environment. During these connections, we ensure the creation of a context handler to pass from the GGML environment to the SECDA environment; the context handler includes memory pointers, memory-mapped model data, access to relevant inputs tensors, quantization, and layer parameters.

## B. SECDA Environment

Within the SECDA components shown in Figure 2, the accelerator designer can start quickly prototyping the initial accelerator design and driver code. First, the user is required to create the initial driver, a simple C++ class that will gain access to the context handler provided by the offload call from GGML . Second, the developer must create an initial SystemC description of their accelerator. Then, the user can instantiate their desired data communication channels between the driver and accelerator using data interfaces provided within the SECDA environment (e.g, AXI4-S, AXI-MM and AXILite). The developer can use these data channels for SystemC end-to-end simulation.

## C. SystemC Simulation

SystemC end-to-end simulation is a crucial step in the SECDA methodology; therefore, SECDA-LLM provides access to SystemC simulation. The simulation-based design loop is shown at the bottom left half of Figure 2. Once the driver and accelerator are connected through the desired data communication channels, the user can perform end-to-end simulations of LLMs using SECDA-LLM. With simulation enabled, the designer can quickly prototype new driver and accelerator features, verifying correctness, profiling performance and modeling control flow behavior within their design. Therefore, the hardware developer is able to rapidly iterate through their design process, through end-to-end simulation, to meet the target performance.

## D. Hardware Evaluation

With simulation-based evaluation, the designer can quickly make fast, broad design changes. Once satisfied with a given design, the designer can quickly take the SystemC-defined design and perform High-level synthesis (HLS) and logic synthesis (HLX) through the hardware synthesis tool provided by SECDA-LLM to map it to the target FPGA, as shown on at bottom right of Figure 2. Additionally, as SECDA-LLM is integrated with the llama.cpp project, we can leverage the llama.cpp project's compilation flow to generate pre-defined applications that use the LLMs through the llama.cpp 's interface. These generated applications will now have complete access to the driver and accelerator for execution on an FPGAenabled device; see Section IV-C for details.

A major benefit of the SECDA methodology, and therefore SECDA-LLM, is that we can reuse the driver and accelerator completely. Therefore, for actual FPGA evaluation the designer does not need to make any changes to the driver to enable real hardware execution, as the SECDA data interfaces switch between simulation and FPGA execution through a simple "SYSC" compiler flag. Once the accelerator is mapped to the target FPGA, the designer can evaluate its performance with their target applications.

## E. Profiling

Through SECDA-LLM, we provide two types of profiling: simulation profiling and execution time profiling. The profiler module shown in Figure 2 highlights how the profiling interacts with both the accelerator design and driver. Additionally, we are able to leverage any additional profiling tools provided by the llama.cpp project.

1) Simulation profiling: End-to-end SystemC simulation can be used to quickly evaluate the potential performance impact of changes to the hardware and software components of the accelerator design and verify the correctness of the implementation. To profile the end-to-end simulation, the developer typically needs to add additional profiling code to keep track of hardware and software metrics throughout the end-to-end LLM inference. The profiler module provided within SECDA-LLM enables the quick and easy method to set up capture points to profile from the accelerator. The capture points can record different metrics of the accelerator and hardware submodules. Metrics include clock cycle counts and the dynamic utilization of processing elements and accelerator buffers.

2) Execution profiling: During the hardware evaluation, SECDA profiling provides execution time for the custom driver and accelerator. This type of profiling helps the designer understand the performance bottlenecks caused by driveraccelerator interactions. For instance, a designer may opt to profile time spent: i) Sending input data; ii) Waiting for the accelerator to execute operations; iii) Unpacking output data received from the accelerator. The analysis of these detailed execution time breakdowns can motivate both accelerator and driver design choices. Additionally, execution profiling of an FPGA evaluation run can be used to profile driver execution times, which can be combined with SystemC-reported simulation times for the accelerator. This would estimate end-to-end execution time in terms of both CPU and accelerator.

## IV. CASE STUDY

To demonstrate our SECDA-LLM platform and how it provides a quick and efficient design flow for developing LLM accelerators for edge devices, we develop a new custom FPGA-based accelerator for BFP quantized LLM inference.

## A. Target Problem

We target the acceleration of MatMul operations within our target LLM, as MatMul represents about 97% of the computations. Specifically, we accelerate the GGML's Mat-Mul\_Q3\_K\_Q8\_K kernel, which uses 3-bit weights and 8bitinputs with BFP quantization.

Both weights and inputs are stored in what is called "superblocks" (SBs); these SBs are critical in maintaining LLM accuracy by adjusting mathematical scaling during computation. With the Q3\_K format used for weights, each SB can represent 256 weights ( N w ), where the SB is partitioned into 16 tiles ( N tiles ) and each tile contains a scaling factor (6-bits) and 16 weights (3-bits). Additionally, each SB has one super-scaling factor (16-bits), which equates to ∼ 3.5 bits-per-weight. With the Q8\_K format, used for inputs, each SB contains 256 inputs (8-bits) and a single super-scaling factor SSF (16-bits), which equates to 8 ∼ bits-per-input.

## B. Accelerator Design

Our accelerator design, shown in Figure 3, contains an instruction decoder , a data mapper , a scheduler and the SuperBlock Vector Processor (SBVP) :

- · The instruction decoder loads and decodes instructions from the AXI-Stream and then communicates the instruction throughout the rest of the accelerator.
- · The data mapper parses the incoming data stream and maps the weight and input super-blocks into their respective weight and input buffers. Our mapping scheme enables efficient data access, so that the SBVP can compute without stalling the computation pipeline.
- · The SBVP efficiently computes the dot product between the SB of weights and inputs while scaling the computation according to the SB scaling factors.
- · The scheduler tiles the MatMul problem according to the dimension of the target layer. Additionally, it synchronizes and accumulates the output data produced by the SBVP and sends the results back to the main memory using the AXI-Stream.

Fig. 3: Overview of our block floating point quantized accelerator design for GGML's MatMul\_Q3\_K\_Q8\_K kernel.

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00462v1-figure-8d0f9a37-5787-4b03-a694-599ca7840dce.jpg)

## C. Evaluation

We evaluate our accelerator design on the PYNQ-Z1 board [15], which contains a Xilinx Z020 edge FPGA and a 650MHz dual-core Arm Cortex-A9 CPU. We execute inference for the TinyLlama model [14], that contains 1.1B parameters (460 MB), trained on the Guanaco dataset [26]. This model contains various BFP quantization levels, but most layers are quantized to Q3\_K . Note that with llama.cpp you can apply different levels of quantization to reduce model size as required.

For our experiments, we use the llama.cpp project's 'main' program cross-compiled for our target CPU architecture, ARMv7a, with Neon vector instructions enabled alongside our accelerator driver. We execute the TinyLlama model utilizing our FPGA-mapped accelerator to offload the Mat-Mul\_Q3\_K\_Q8\_K layers, to obtain an initial speed of 1 7 . seconds per token ( ∼ 2 seconds per word). This provides a 11 × speedup over CPU-only inference, drastically improving the usability of LLMs on such a resource-constraint device. Therefore, with SECDA-LLM we could quickly implement, integrate and evaluate our new accelerator design.

## V. CONCLUSION

We introduced SECDA-LLM, a novel platform that simplifies the creation of specialized hardware accelerators for LLM inference on resource-constrained edge devices. SECDA-LLM integrates the SECDA methodology within llama.cpp , enabling developers to access the SECDA tools (e.g., AXIAPI, profiler), which can be used to effectively co-design new FPGA-based accelerators for LLMs with ease. As a case study, we presented a quantized MatMul accelerator design that optimizes LLM inference (by 11 × over the dualcore Arm NEON-based CPU execution) for the TinyLlama model. Future work will expand SECDA-LLM into an opensource platform for collaborative development and continuous improvement of LLMs' performance on resource-constrained edge devices.

## ACKNOWLEDGMENT

This work was partially supported by the Engineering and Physical Sciences Research Council (grant EP/R513222/1) and the EU Project dAIEDGE (GA Nr 101120726).

## REFERENCES

- [1] T. B. Brown et al. Language Models are Few-Shot Learners. [Online]. Available: http://arxiv.org/abs/2005.14165
- [2] P. P. Ray, 'ChatGPT: A comprehensive review on background, applications, key challenges, bias, ethics, limitations and future scope,' vol. 3, pp. 121-154. [Online]. Available: https://www.sciencedirect. com/science/article/pii/S266734522300024X
- [3] B. Yao et al. Benchmarking LLM-based Machine Translation on Cultural Awareness. [Online]. Available: http://arxiv.org/abs/2305.14328
- [4] X. Sun et al. Text Classification via Large Language Models. [Online]. Available: http://arxiv.org/abs/2305.08377
- [5] J. Liu et al. , 'Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation,' vol. 36, pp. 21 558-21 572. [Online]. Available: https://proceedings.neurips.cc/paper\_files/paper/2023/hash/ 43e9d647ccd3e4b7b5baab53f0368686-Abstract-Conference.html
- [6] J. Clusmann et al. , 'The future landscape of large language models in medicine,' vol. 3, no. 1, pp. 1-8. [Online]. Available: https://www.nature.com/articles/s43856-023-00370-1
- [7] 'GGML,' https://github.com/ggerganov/ggml, 2024.
- [8] 'llama.cpp,' https://github.com/ggerganov/llama.cpp, 2024.
- [9] H. Khan et al. , 'NPE: An FPGA-based Overlay Processor for Natural Language Processing,' in The 2021 ACM/SIGDA International Symposium on Field-Programmable Gate Arrays (FPGA '21) . New York, NY, USA: Association for Computing Machinery, Feb. 2021, p. 227.
- [10] S. Lu et al. , 'Hardware Accelerator for Multi-Head Attention and Position-Wise Feed-Forward in the Transformer,' in 2020 IEEE 33rd International System-on-Chip Conference (SOCC) . Las Vegas, NV, USA: IEEE, Sep. 2020, pp. 84-89.
- [11] J. Haris et al. , 'SECDA: Efficient Hardware/Software Co-Design of FPGA-based DNN Accelerators for Edge Inference,' in 2021 IEEE 33rd International Symposium on Computer Architecture and High Performance Computing (SBAC-PAD) , pp. 33-43. [Online]. Available: https://ieeexplore.ieee.org/document/9651579
- [12] J. Haris et al. , 'SECDA-TFLite: A toolkit for efficient development of FPGA-based DNN accelerators for edge inference,' vol. 173, pp. 140-151. [Online]. Available: https://www.sciencedirect.com/science/ article/pii/S0743731522002301
- [13] P. Gibson et al. , 'DLAS: A Conceptual Model for Across-Stack Deep Learning Acceleration,' ACM Transactions on Architecture and Code Optimization (TACO) , 2024.
- [14] P. Zhang et al. TinyLlama: An Open-Source Small Language Model. [Online]. Available: http://arxiv.org/abs/2401.02385
- [15] 'PYNQ Z1 Manual,' https://reference.digilentinc.com/reference/ programmable-logic/pynq-z1/reference-manual, 2020.
- [16] A. Vaswani et al. , 'Attention is all you need,' Advances in neural information processing systems , vol. 30, 2017.
- [17] H. Touvron et al. , 'Llama: Open and efficient foundation language models,' arXiv preprint arXiv:2302.13971 , 2023.
- [18] W. Kwon et al. , 'Efficient memory management for large language model serving with pagedattention,' in Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles , 2023.
- [19] O. Zafrir et al. , 'Q8BERT: Quantized 8Bit BERT,' in 2019 Fifth Workshop on Energy Efficient Machine Learning and Cognitive Computing NeurIPS Edition (EMC2-NIPS) . IEEE Computer Society, Dec. 2019, pp. 36-39.
- [20] L. J. Wan et al. , 'Software/Hardware Co-design for LLM and Its Application for Design Verification,' in 2024 29th Asia and South Pacific Design Automation Conference (ASP-DAC) . Incheon, Korea, Republic of: IEEE, Jan. 2024, pp. 435-441.
- [21] X. Shen et al., 'EdgeQAT: Entropy and Distribution Guided Quantization-Aware Training for the Acceleration of Lightweight LLMs on the Edge,' http://arxiv.org/abs/2402.10787, 2024, accessed: Apr. 18, 2024.
- [22] B. D. Rouhani et al., 'Microscaling Data Formats for Deep Learning,' https://arxiv.org/abs/2310.10537, 2023, accessed: October 19, 2023.
- [23] Simon Willison, 'Alpaca: A New Programming Language for Data Analysis,' https://simonwillison.net/2023/Mar/13/alpaca/, 2023, accessed: March 13, 2023.
- [24] C. Kachris, 'A Survey on Hardware Accelerators for Large Language Models,' http://arxiv.org/abs/2401.09890, 2024, accessed: Apr. 17, 2024.
- [25] IEEE, 'IEEE Standard for Standard SystemC Language Reference Manual,' IEEE Std 1666-2011 , 2012.
- [26] Joséphus Cheung, 'Guanacodataset (revision 892e57a),' 2023. [Online]. Available: https://huggingface.co/datasets/JosephusCheung/ GuanacoDataset
Jude Haris Rappy Saha Wenhao Hu José Cano
Designing Efficient LLM Accelerators for Edge Devices
The increase in open-source availability of Large Language Models (LLMs) has
enabled users to deploy them on more and more resource-constrained edge devices
to reduce reliance on network connections and provide more privacy. However,
the high computation and memory demands of LLMs make their execution on
resource-constrained edge devices challenging and inefficient. To address this
issue, designing new and efficient edge accelerators for LLM inference is
crucial. FPGA-based accelerators are ideal for LLM acceleration due to their
reconfigurability, as they enable model-specific optimizations and higher
performance per watt. However, creating and integrating FPGA-based accelerators
for LLMs (particularly on edge devices) has proven challenging, mainly due to
the limited hardware design flows for LLMs in existing FPGA platforms.
  To tackle this issue, in this paper we first propose a new design platform,
named SECDA-LLM, that utilizes the SECDA methodology to streamline the process
of designing, integrating, and deploying efficient FPGA-based LLM accelerators
for the llama.cpp inference framework. We then demonstrate, through a case
study, the potential benefits of SECDA-LLM by creating a new MatMul accelerator
that supports block floating point quantized operations for LLMs. Our initial
accelerator design, deployed on the PYNQ-Z1 board, reduces latency 1.7 seconds
per token or ~2 seconds per word) by 11x over the dual-core Arm NEON-based CPU
execution for the TinyLlama model.