Transformer-based language models such as BERT provide significant accuracy improvement to a multitude of natural language processing (NLP) tasks. However, their hefty computational and memory demands make them challenging to deploy to resource-constrained edge platforms with strict latency requirements. We present EdgeBERT  an in-depth and principled algorithm and hardware design methodology to achieve minimal latency and energy consumption on multi-task NLP inference. Compared to the ALBERT baseline, we achieve up to 2.4x and 13.4x inference latencyand memory savings, respectively, with less than 1\%-pt drop in accuracy on several GLUE benchmarks by employing a calibrated combination of 1) entropy-based early stopping, 2) adaptive attention span, 3) movement and magnitude pruning, and 4) floating-point quantization. Furthermore, in order to maximize the  benefits of these algorithms in always-on and intermediate edge computing settings, we specialize a scalable hardware architecture wherein floating-point bit encodings of the shareable multi-task embedding parameters are stored in high-density non-volatile memory . Altogether, EdgeBERT enables fully on-chip inference acceleration of NLP workloads with 5.2 x, and 157x lower energy than that of an un-optimized accelerator and CUDA adaptations on an Nvidia Jetson Tegra X2 mobile GPU, respectively.
Transformer-based language models such as BERT provide significant accuracy improvement for a multitude of natural language processing (NLP) tasks. However, their hefty computational and memory demands make them challenging to deploy to resource-constrained edge platforms with strict latency requirements. We present EdgeBERT , an in-depth algorithm-hardware co-design for latency-aware energy optimization for multi-task NLP . EdgeBERT employs entropy-based early exit predication in order to perform dynamic voltage-frequency scaling (DVFS), at a sentence granularity, for minimal energy consumption while adhering to a prescribed target latency. Computation and memory footprint overheads are further alleviated by employing a calibrated combination of adaptive attention span, selective network pruning, and floating-point quantization. Furthermore, in order to maximize the synergistic benefits of these algorithms in always-on and intermediate edge computing settings, we specialize a 12nm scalable hardware accelerator system, integrating a fast-switching low-dropout voltage regulator (LDO), an all-digital phase-locked loop (ADPLL), as well as, high-density embedded non-volatile memories (eNVMs) wherein the sparse floating-point bit encodings of the shared multi-task parameters are carefully stored . Altogether, latency-aware multi-task NLP inference acceleration on the EdgeBERT hardware system generates up to 7x, 2.5 x, and 53x lower energy compared to the conventional inference without early stopping, the latency-unbounded early exit approach, and CUDA adaptations on an Nvidia Jetson Tegra X2 mobile GPU, respectively.
R
to
for
A
,
R
and principled algorithm and hardware design methodology to achieve minimal latency and energy consumption on
algorithm-hardware co-design for latency-aware energy optimization for
R
inference. Compared to the ALBERT baseline, we achieve up to 2.4x and 13.4x inference latencyand memory savings, respectively, with less than 1\%-pt drop in accuracy on several GLUE benchmarks
. EdgeBERT employs entropy-based early exit predication in order to perform dynamic voltage-frequency scaling (DVFS), at a sentence granularity, for minimal energy consumption while adhering to a prescribed target latency. Computation and memory footprint overheads are further alleviated
D
1) entropy-based early stopping, 2)
R
3) movement and magnitude
selective network
D
4)
A
synergistic
R
scalable hardware architecture wherein
12nm scalable hardware accelerator system, integrating a fast-switching low-dropout voltage regulator (LDO), an all-digital phase-locked loop (ADPLL), as well as, high-density embedded non-volatile memories (eNVMs) wherein the sparse
R
shareable
shared
R
embedding parameters are stored in high-density non-volatile memory
parameters are carefully stored
R
EdgeBERT enables fully on-chip inference acceleration of NLP workloads with 5.2
latency-aware multi-task NLP inference acceleration on the EdgeBERT hardware system generates up to 7x, 2.5
R
157x lower energy than that of an un-optimized accelerator
53x lower energy compared to the conventional inference without early stopping, the latency-unbounded early exit approach,