  It is appealing but challenging to achieve real-time deep neural network
(DNN) inference on mobile devices because even the powerful modern mobile
devices are considered as ``resource-constrained'' when executing large-scale
DNNs. It necessitates the sparse model inference via weight pruning, i.e., DNN
weight sparsity, and it is desirable to design a new DNN weight sparsity scheme
that can facilitate real-time inference on mobile devices while preserving a
high sparse model accuracy. This paper designs a novel mobile inference
acceleration framework GRIM that is General to both convolutional neural
networks (CNNs) and recurrent neural networks (RNNs) and that achieves
Real-time execution and high accuracy, leveraging fine-grained structured
sparse model Inference and compiler optimizations for Mobiles. We start by
proposing a new fine-grained structured sparsity scheme through the Block-based
Column-Row (BCR) pruning. Based on this new fine-grained structured sparsity,
our GRIM framework consists of two parts: (a) the compiler optimization and
code generation for real-time mobile inference; and (b) the BCR pruning
optimizations for determining pruning hyperparameters and performing weight
pruning. We compare GRIM with Alibaba MNN, TVM, TensorFlow-Lite, a sparse
implementation based on CSR, PatDNN, and ESE (a representative FPGA inference
acceleration framework for RNNs), and achieve up to 14.08x speedup.

Machine Learning;Artificial Intelligence