  The widespread application of artificial neural networks has prompted
researchers to experiment with FPGA and customized ASIC designs to speed up
their computation. These implementation efforts have generally focused on
weight multiplication and signal summation operations, and less on activation
functions used in these applications. Yet, efficient hardware implementations
of nonlinear activation functions like Exponential Linear Units (ELU), Scaled
Exponential Linear Units (SELU), and Hyperbolic Tangent (tanh), are central to
designing effective neural network accelerators, since these functions require
lots of resources. In this paper, we explore efficient hardware implementations
of activation functions using purely combinational circuits, with a focus on
two widely used nonlinear activation functions, i.e., SELU and tanh. Our
experiments demonstrate that neural networks are generally insensitive to the
precision of the activation function. The results also prove that the proposed
combinational circuit-based approach is very efficient in terms of speed and
area, with negligible accuracy loss on the MNIST, CIFAR-10 and IMAGENET
benchmarks. Synopsys Design Compiler synthesis results show that circuit
designs for tanh and SELU can save between 3.13-7.69 and 4.45-8:45 area
compared to the LUT/memory-based implementations, and can operate at 5.14GHz
and 4.52GHz using the 28nm SVT library, respectively. The implementation is
available at: https://github.com/ThomasMrY/ActivationFunctionDemo.

Neural and Evolutionary Computing