IoT Edge intelligence requires Convolutional Neural Network (CNN) inference to take place in the edge device itself. ARM big.LITTLE architecture is at the heart of common commercial edge devices. It comprises of single-ISA heterogeneous multi-cores grouped in homogeneous clusters that enables performance and power trade-offs.  However, high communication overhead involved in parallelization of computation from a convolution kernel across clusters is detrimental to throughput. We present an alternative framework called Pipe-it that employs a pipelined design to split the convolutional layers across clusters while limiting the parallelization of their respective kernels to the assigned clusters . We develop a performance prediction model that , from convolutional layer descriptors , predicts the execution time of each layer individually on all different core types and number of cores . Pipe-it then exploits the predictions to create a balanced pipeline using an efficient design space exploration algorithm. Pipe-it on average results in  39\% higher throughput than the highest antecedent throughput.
IoT Edge intelligence requires Convolutional Neural Network (CNN) inference to take place in the edge devices itself. ARM big.LITTLE architecture is at the heart of prevalent commercial edge devices. It comprises of single-ISA heterogeneous cores grouped into multiple homogeneous clusters that enable power and performance trade-offs. All cores are expected to be simultaneously employed in inference to attain maximal throughput. However, high communication overhead involved in parallelization of computations from convolution kernels across clusters is detrimental to throughput. We present an alternative framework called Pipe-it that employs pipelined design to split convolutional layers across clusters while limiting parallelization of their respective kernels to the assigned cluster . We develop a performance-prediction model that utilizes only the convolutional layer descriptors to predict the execution time of each layer individually on all permitted core configurations (type and count) . Pipe-it then exploits the predictions to create a balanced pipeline using an efficient design space exploration algorithm. Pipe-it on average results in a 39\% higher throughput than the highest antecedent throughput.
R
device
devices
R
common
prevalent
R
multi-cores grouped in
cores grouped into multiple
R
enables performance and power
enable power and performance
A
All cores are expected to be simultaneously employed in inference to attain maximal throughput.
R
computation from a convolution kernel
computations from convolution kernels
D
a
D
the
D
the
R
clusters
cluster
R
performance prediction model that , from
performance-prediction model that utilizes only the
R
, predicts
to predict
R
different core types and number of cores
permitted core configurations (type and count)
A
a