WORKS BY EXPLOITING NUMERICAL PRECISION VARIABILITY
Tartan TRT a hardware accelerator for inference with Deep Neural Networks (DNNs) is presented and evaluated on Convolutional Neural Networks. TRT exploits the variable per layer precision requirements of DNNs to deliver execution time that is proportional to the precision p in bits used per layer for convolutional and fully-connected layers. Prior art has demonstrated an accelerator with the same execution performance only for convolutional layersJudd et al. (2016a;c). Experiments on image classification CNNs show that on average across all networks studied, TRT outperforms a state-of-the-art bit-parallel accelerator Chen et al. (2014b) by 1.90× without any loss in accuracy while it is 1.17× more energy efficient. TRT requires no network retraining while it enables trading off accuracy for additional improvements in execution performance and energy efficiency. For example, if a 1% relative loss in accuracy is acceptable, TRT is on average 2.04× faster and 1.25× more energy efficient than a conventional bitparallel accelerator. A Tartan configuration that processes 2-bits at time, requires less area than the 1-bit configuration, improves efficiency to 1.24× over the bitparallel baseline while being 73% faster for convolutional layers and 60% faster for fully-connected layers is also presented.
1 INTRODUCTION
It is only recently that commodity computing hardware in the form of graphics processors delivered the performance necessary for practical, large scale Deep Neural Network applications Krizhevsky et al. (2012). At the same time, the end of Dennard Scaling in semiconductor technology Esmaeilzadeh et al. (2011) makes it difficult to deliver further advances in hardware performance using existing general purpose designs. It seems that further advances in DNN sophistication would have to rely mostly on algorithmic and in general innovations at the software level which can be helped by innovations in hardware design. Accordingly, hardware DNN accelerators have emerged. The DianNao accelerator family was the first to use a wide single-instruction single-data (SISD) architecture to process up to 4K operations in parallel on a single chip Chen et al. (2014a;b) outperforming graphics processors by two orders of magnitude. Development in hardware accelerators has since proceeded in two directions: either toward more general purpose accelerators that can support more machine learning algorithms while keeping performance mostly on par with DaDianNao (DaDN) Chen et al. (2014b), or toward further specialization of specific layers or classes of DNNs with the goal of outperforming DaDN in execution time and/or energy efficiency, e.g., Han et al. (2016); Albericio et al. (2016a); Judd et al. (2016a); Chen, Yu-Hsin and Krishna, Tushar and Emer, Joel and Sze, Vivienne (2016); Reagen et al. (2016). This work is along the second direction. Section 5 reviews several other accelerator designs.
While DaDN’s functional units process 16-bit fixed-point values, DNNs exhibit varying precision requirements across and within layers, e.g., Judd et al. (2015). Accordingly, it is possible to use
shorter, per layer representations for activations and/or weights. However, with existing bit-parallel functional units doing so does not translate into a performance nor an energy advantage as the values are expanded into the native hardware precision inside the unit.
This work presents Tartan (TRT), a massively parallel hardware accelerator whose execution time for fully-connected and convolutional layers scales with the precision p used to represent the input values. TRT uses hybrid bit-serial/bit-parallel functional units and exploits the abundant parallelism of typical DNN layers with the goal of exceeding DaDN’s execution time performance and energy efficiency. Ideally Tartan can improve execution time by 16p where p is the precision used for the activations in convolutional layers, and for the activations and weights in fully-connected layers. Every bit of precision that can be eliminated ideally reduces execution time and increases energy efficiency. TRT builds upon the Stripes (STR) accelerator Judd et al. (2016c;a) which improves execution time and energy efficiency only for convolutional layers.
This work evaluates TRT on a set of convolutional neural networks (CNNs) for image classification. On average TRT reduces inference time by 1.61×, 1.91× and 1.90× over DaDN for the fullyconnected, the convolutional, and all layers respectively. Energy efficiency compared to DaDN with TRT is 0.92×, 1.18× and 1.17× respectively. TRT enables trading off accuracy for improving execution time and energy efficiency. For example, on average for the fully-connected layers, accepting a 1% loss in accuracy improves performance to 1.73× and energy efficiency to 1.00× compared to DaDN.
The rest of this document is organized as follows: Section 2 illustrates the key concepts behind TRT via an example. Section 3 reviews the DaDN architecture and presents an equivalent Tartan configuration. Section 4 presents the experimental results. Section 5 reviews related work and discusses the limitations of this study and the potential challenges with TRT . Section 6 concludes.
2 Tartan: A SIMPLIFIED EXAMPLE
This section illustrates at a high-level the TRT design by showing how it would process two purposely trivial cases: 1) a fully-connected layer (FCL) with a single input activation producing two output activations, and 2) a convolutional layer (CVL) with two input activations and one singleweight filter producing two output activations. The per layer calculations are:
Fully − Connected : Convolutional : f1 = w1 × a c1 = w × a1 f2 = w2 × a c2 = w × a2
Where f1, f2, c1 and c2 are output activations, w1, w2, and w are weights, and a1, a2 and a are input activations. For clarity all values are assumed to be represented in 2 bits of precision.
2.1 CONVENTIONAL BIT-PARALLEL PROCESSING
Figure 2.1a shows a bit-parallel processing engine representative of DaDN. Every cycle, the engine can calculate the product of two 2-bit inputs, i (weight) and v (activation) and accumulate or store it into the output register OR. Parts (b) and (c) of the figure show how this unit can calculate the example CVL over two cycles. In part (b) and during cycle 0, the unit accepts along the v input bits 0 and 1 of a1 (noted as a1/0 and a1/1 respectively on the figure), and along i bits 0 and 1 of w and produces both bits of output c1. Similarly, during cycle 1 (part (c)), the unit processes a2 and w to produce c2. In total, over two cycles, the engine produced two 2b × 2b products. Processing the example FCL also takes two cycles: In the first cycle w1 and a produce f1, and in the second cycle w2 and a produce f2. This process is not shown in the interest of space.
2.2 Tartan’S APPROACH
Figure 2 shows how a TRT-like engine would process the example CVL. Figure 2a shows the engine’s structure which comprises two subunits. The two subunits accept each one bit of an activation per cycle through inputs v0 and v1 respectively and as before, there is a common 2-bit weight input (i1, i0). In total, the number of input bits is 4, identical to the bit-parallel engine.
Each subunit contains three 2-bit registers: a shift-register AR, a parallel load register BR, and an parallel load output register OR. Each cycle each subunit can calculate the product of its single bit vi input with BR which it can write or accumulate into its OR. There is no bit-parallel multiplier since the subunits process a single activation bit per cycle. Instead, two AND gates, a shift-and-add functional unit, and OR form a shift-and-add multiplier/accumulator. Each AR can load a single bit per cycle from one of the i wires, and BR can be parallel loaded from AR or from the i wires.
Convolutional Layer: Figure 2b through Figure 2d show how the CVL is processed. The figures abstract away the unit details showing only the register contents. As Figure 2b shows, during cycle 1, the w synapse is loaded in parallel to the BRs of both subunits via the i1 and i0 inputs. During cycle 2, bits 0 of a1 and of a2 are sent via the v0 and v1 inputs respectively to the first and second subunit. The subunits calculate concurrently a1/0 × w and a2/0 × w and accumulate these results into their ORs. Finally, in cycle 3, bit 1 of a1 and a2 appear respectively on v0 and v1. The subunits calculate respectively a1/1 × w and a2/1 × w accumulating the final output activations c1 and c2 into their ORs.
In total it took 3 cycles to process the layer. However, at the end of the third cycle, another w could have been loaded into the BRs (the i are idle) allowing a new set of outputs to commence computation during cycle 4. That is loading a new weight can be hidden during the processing of the current output activation for all but the first time. In the steady state, when the input activations are represented in two bits, this engine will be producing two 2b × 2b terms every two cycles thus matching the bandwidth of the bit-parallel engine.
If the activations a1 and a2 could be represented in just one bit, then this engine would be producing two output activations per cycle, twice the bandwidth of the bit-parallel engine. The latter is incapable of exploiting the reduced precision. In general, if the bit-parallel hardware was using Pbase bits to represent the activations while only Pa bits were enough, TRT would outperform the bit-parallel engine by PbasePTRT .
Fully-Connected Layer: Figure 3 shows how a TRT-like unit would process the example FCL. As Figure 3a shows, in cycle 1, bit 1 of w1 and of w2 appear respectively on lines i1 and i0. The left subunit’s AR is connected to i1 while the right subunit’s AR is connected to i0. The ARs shift in the corresponding bits into their least significant bit sign-extending to the vacant position (shown as a 0 bit on the example). During cycle 2, as Figure 3b shows, bits 0 of w1 and of w2 appear on the respective i lines and the respective ARs shift them in. At the end of the cycle, the left subunit’s AR contains the full 2-bit w1 and the right subunit’s AR the full 2-bit w2. In cycle 3, Figure 3c shows that the contents of AR are copied to BR in each subunit. From the next cycle, calculating the products can now proceed similarly to what was done for the CVL. In this case, however, each BR contains a different weight whereas in the CVL all BRs held the same w value. The shift capability of the ARs coupled with the different i wire per subunit connection allowed us to load a different weight bit-serially over two cycles. Figure 3d and Figure 3e show cycles 4 and 5 respectively. During cycle 4, bit 0 of a1 appears on both v inputs and is multiplied with the BR in each subunit. In cycle 5, bit 1 of a1 appears on both v inputs and the subunits complete the calculation of f1 and f2. It takes two cycles to produce the two 2b× 2b products once the correct inputs appear into the BRs. While in our example no additional inputs nor outputs are shown, it would have been possible to overlap the loading of a new set of w inputs into the ARs while processing the current weights stored into the BRs. That is the loading into ARs, copying into BRs, and the bit-serial multiplication of the BRs with the activations is a 3-stage pipeline where each stage can take multiple cycles. In general, assuming that both activations and weights are represented using 2 bits, this engine would match the performance of the bit-parallel engine in the steady state. When both set of inputs i and v can be represented with fewer bits, 1 in this case, the engine would produce two terms per cycle, twice the bandwidth of the bit-parallel engine of the previous section.
Summary: In general, if Pbase the precision of the bit-parallel engine, and PLa and PLw the precisions that can be used respectively for activations and weights for layer L, a TRT engine can ideally outperform an equivalent bit parallel engine by Pbase
PLa for CVLs, and by Pbase max(PLa ,P L w ) for FCLs. This example used the simplest TRT engine configuration. Since typical layers exhibit massive parallelism, TRT can be configured with many more subunits while exploiting weight reuse for CVLs and activation reuse for FCLs. The next section describes the baseline state-of-the-art DNNs accelerator and presents an equivalent TRT configuration.
3 Tartan ARCHITECTURE
This work presents TRT as a modification of the state-of-the-art DaDianNao accelerator. Accordingly, Section 3.1 reviews DaDN’s design and how it can process FCLs and CVLs. For clarity, in what follows the term brick refers to a set of 16 elements of a 3D activation or weight array1 input which are contiguous along the i dimension, e.g., a(x, y, i)...a(x, y, i+ 15). Bricks will be denoted by their origin element with a B subscript, e.g., aB(x, y, i). The size of a brick is a design parameter.
3.1 BASELINE SYSTEM: DADIANNAO
TRT is demonstrated as a modification of the DaDianNao accelerator (DaDN) proposed by Chen et al. (2014b). Figure 4a shows a DaDN tile which processes 16 filters concurrently calculating 16 activation and weight products per filter for a total of 256 products per cycle. Each cycle the tile accepts 16 weights per filter for total of 256 synapses and 16 input activations. The tile multiplies each weight with only one activation whereas each activation is multiplied with 16 weights, one per filter. The tile reduces the 16 products into a single partial output activation per filter, for a total of 16 partial output activations for the tile. Each DaDN chip comprises 16 such tiles, each processing a different set of 16 filters per cycle. Accordingly, each cycle, the whole chip processes 16 activations and 256× 16 = 4K weights producing 16× 16 = 256 partial output activations, 16 per tile. Internally, each tile has: 1) a synapse buffer (SB) that provides 256 weights per cycle one per weight lane, 2) an input neuron buffer (NBin) which provides 16 activations per cycle through 16 neuron lanes, and 3) a neuron output buffer (NBout) which accepts 16 partial output activations per cycle. In the tile’s datapath each activation lane is paired with 16 weight lanes one from each filter. Each synapse and neuron lane pair feeds a multiplier, and an adder tree per filter lane reduces the 16 per filter products into a partial sum. In all, the filter lanes produce each a partial sum per cycle, for a
1An FCL can be thought of as a CVL where the input activation array has unit x and y dimensions, and there are as many filters as output activations, and where the filter dimenions are identical to the input activation array.
total of 16 partial output activations per Once a full window is processed, the 16 resulting sums, are fed through a non-linear activation function, f , to produce the 16 final output activations. The multiplications and reductions needed per cycle are implemented via 256 multipliers one per weight lane and sixteen 17-input (16 products plus the partial sum from NBout) adder trees one per filter lane.
Figure 5a shows an overview of the DaDN chip. There are 16 processing tiles connected via an interconnect to a shared central eDRAM Neuron Memory (NM). DaDN’s main goal was minimizing off-chip bandwidth while maximizing on-chip compute utilization. To avoid fetching weights from off-chip, DaDN uses a 2MB eDRAM Synapse Buffer (SB) for weights per tile for a total of 32MB eDRAM. All inter-layer activation outputs except for the initial input and the final output are stored in NM which is connected via a broadcast interconnect to the 16 Input Neuron Buffers (NBin) buffers. All values are 16-bit fixed-point, hence a 256-bit wide interconnect can broadcast a full activation brick in one step. Off-chip accesses are needed only for reading: 1) the input image, 2) the weight once per layer, and 3) for writing the final output.
Processing starts by reading from external memory the first layer’s filter weights, and the input image. The weights are distributed over the SBs and the input is stored into NM. Each cycle an input activation brick is broadcast to all units. Each units reads 16 weight bricks from its SB and produces a partial output activation brick which it stores in its NBout. Once computed, the output activations are stored through NBout to NM and then fed back through the NBins when processing the next layer. Loading the next set of weights from external memory can be overlapped with the processing of the current layer as necessary.
3.2 Tartan
As Section 2 explained, TRT processes activations bit-serially multiplying a single activation bit with a full weight per cycle. Each DaDN tile multiplies 16 16-bit activations with 256 weights each cycle. To match DaDN’s computation bandwidth, TRT needs to multiply 256 1-bit activations with 256 weights per cycle. Figure 4b shows the TRT tile. It comprises 256 Serial Inner-Product Units (SIPs) organized in a 16×16 grid. Similar to DaDN each SIP multiplies 16 weights with 16 activations and reduces these products into a partial output activation. Unlike DaDN, each SIP accepts 16 single-bit activation inputs. Each SIP has two registers, each a vector of 16 16-bit subregisters: 1) the Serial Weight Register (SWR), and 2) the Weight Register (WR). These correspond to AR and BR of the example of Section 2. NBout remains as in DaDN, however, it is distributed along the SIPs as shown.
Convolutional Layers: Processing starts by reading in parallel 256 weights from the SB as in DaDN, and loading the 16 per SIP row weights in parallel to all SWRs in the row. Over the next PLa cycles, the weights are multiplied by the bits of an input activation brick per column. TRT exploits weight reuse across 16 windows sending a different input activation brick to each column. For example, for a CVL with a stride of 4 a TRT tile will processes 16 activation bricks aB(x, y, i), aB(x+ 4, y, i) through aB(x+ 63, y, i) in parallel a bit per cycle. Assuming that the tile processes filters fi though fi+15, after PLa cycles it would produce the following partial output activations: oB(x/4, y/4, fi), through oB(x/4 + 15, y/4, fi), that is 16 contiguous on the x dimension output activation bricks. Whereas DaDN would process 16 activations bricks over 16 cycles, TRT processes them concurrently but bit-serially over PLa cycles. If P L a is less than 16, TRT will outperform DaDN by 16/PLa , and when P L a is 16, TRT will match DaDN’s performance.
Fully-Connected Layers: Processing starts by loading bit-serially and in parallel over PLw cycles, 4K weights into the SWRs. Each SWR per row gets a different set of 16 weights as each subregister is connected to one out of the 256 wires of the SB output bus for the SIP row. Once the weights have been loaded, the SWRs are copied to the SWs and multiplication with the input activations can then proceed bit-serially over PLa cycles. Assuming that there are enough output activations so that a different output activation can be assigned to each SIP, the same input activation brick can be broadcast to all SIP columns. For example, for an FCL a TRT tile will process one activation brick aB(i) bit-serially to produce 16 output activation bricks oB(i) through oB(i×16) one per SIP column. Loading the next set of weights can be done in parallel with processing the current set, thus execution time is constrained by PLmax = max(P L a , P L w ). Thus, a TRT tile produces 256 partial
output activations every PLmax cycles, a speedup of 16/Pmax over DaDN since a DaDN tile always needs 16 cycles to do the same.
For TRT to be fully utilized an FCL must have at least 4K output activations. Some of the networks studied have a layer with as little as 2K output activations. To avoid underutilization, the SIPs along each row are cascaded into a daisy-chain, where the output of one can feed into an input of the next via a multiplexer. This way, the computation of an output activation can be sliced over the SIPs along the same row. In this case, each SIP processes only a portion of the input activations resulting into several partial output activations along the SIPs on the same row. Over the next np cycles, where np the number of slices used, the np partial outputs can be reduced into the final output activation. The user can chose any number of slices up to 16, so that TRT can be fully utilized even with fullyconnected layers of just 256 outputs. For example, in NeuralTalk Karpathy & Li (2014) the smallest layers can have 600 outputs or fewer.
Other Layers: TRT like DaDN can process the additional layers needed by the studied networks. For this purpose the tile includes additional hardware support for max pooling similar to DaDN. An activation function unit is present at the output of NBout in order to apply nonlinear activations before the output neurons are written back to NM.
3.3 SIP AND OTHER COMPONENTS
SIP: Bit-Serial Inner-Product Units: Figure 6 shows TRT’s Bit-Serial Inner-Product Unit (SIP). Each SIP multiplies 16 activations by 16 weights to produce an output activation. Each SIP has two registers, a Serial Weight Register (SWR) and a Weight Registers (WR), each containing 16 16-bit subregisters. Each SWR subregister is a shift register with a single bit connection to one of the weight bus wires that is used to read weights bit-serially for FCLs. Each WR subregister can be parallel loaded from either the weight bus or the corresponding SWR subregister, to process CVLs or FCLs respectively. Each SIP includes 256 2-input AND gates that multiply the weights in the WR with the incoming activation bits, and a 16 × 16b adder tree that sums the partial products. A final adder plus a shifter accumulate the adder tree results into an output register. In each SIP, a multiplexer at the first input of the adder tree implements the cascade mode supporting slicing the output activation computation along the SIPs of a single row. To support signed 2’s complement neurons, the SIP can subtract the weight corresponding to the most significant bit (MSB) from the partial sum when the MSB is 1. This is done with negation blocks for each weight before the adder tree. Each SIP also includes a comparator (max) to support max pooling layers.
Dispatcher and Reducers: Figure 5b shows an overview of the full TRT system. As in DaDN there is a central NM and 16 tiles. A Dispatcher unit is tasked with reading input activations from NM always performing eDRAM-friendly wide accesses. It transposes each activation and communicates each a bit a time over the global interconnect. For CVLs the dispatcher has to maintain a pool of multiple activation bricks, each from different window, which may require fetching multiple rows from NM. However, since a new set of windows is only needed every PLa cycles, the dispatcher can keep up for the layers studied. For FCLs one activation brick is sufficient. A Reducer per title is tasked with collecting the output activations and writing them to NM. Since output activations take multiple cycles to produce, there is sufficient bandwidth to sustain all 16 tiles.
3.4 PROCESSING SEVERAL BITS AT ONCE
In order to improve TRT’s area and power efficiency, the number of bits processed at once can be parameterized. In this case, the weights are multiplied with several activation bits at once, and the multiplication results are partially shifted before they are inserted into their corresponding adder tree.
In order to load the weights on time, the SWR subregister has to be modified so it can load several bits in parallel, and shift that number of positions every cycle. The negation block (for 2’s complement support) will operate only over the most significant product result.
The chief advantage of such a design is that less SIPs are needed in order to achieve the same throughput – for example, processing 2 bits at once allows reducing the number of columns from 16 to 8. Although the total number of bus wires is similar, the distance they have to cover is significantly reduced. Likewise, the total number of adders required stays similar, but they are clustered closer together.
A drawback of this design is the limitation to precisions that are exact multiples of the number of bits processed at once.
4 EVALUATION
This section evaluates TRT’s performance, energy and area and explores the trade-off between accuracy and performance comparing to DaDN.
4.1 METHODOLOGY
Numerical Representation Requirements Analysis: The per layer precision profiles are found via the methodology of Judd et al. Judd et al. (2015). Caffe Jia et al. (2014) was used to measure how reducing the precision of each FCL affects the network’s overall top-1 prediction accuracy over 5000 images. The network definitions and pre-trained synaptic weights are taken from the Caffe Model Zoo Jia (2015). Since TRT’s performance for FCLs is bound by the maximum of the weight and activation precisions, our exploration was limited to the cases where both are the same. The search procedure is a gradient descent where a given layer’s precision is iteratively decremented one bit at a time, until the network’s accuracy drops. For weights, the fixed point numbers are set to represent values between -1 and 1. For activations, the number of fractional bits is fixed to a previouslydetermined value known not to hurt accuracy, as per Judd et al. (2015). While both activations and weights use the same number of bits, their precisions and ranges differ.
Performance, Area and Energy: DaDN, STR and TRT were modeled using the same methodology for consistency. A custom cycle-accurate simulator models execution time. Computation was scheduled as described by Judd et al. (2016a) to maximize energy efficiency for DaDN. The logic components of the both systems were synthesized with the Synopsys Design Compiler Synopsys for a TSMC 65nm library to report power and area. The circuit is clocked at 980 MHz. The NBin and NBout SRAM buffers were modelled using CACTI Muralimanohar & Balasubramonian. The eDRAM area and energy were modelled with Destiny Poremba et al. (2015).
4.2 RESULTS
Fully-Connected Layer Precisions: Table 1 reports the per layer precisions for the CVLs and FCLs of the networks studied along with the speedup over DaDN that would be ideally possible. The discussion in this section focuses solely on FCLs. The precisions that can be used vary from 8 up to 10 bits vs. the 16 bits DaDN uses. The ideal speedup ranges from 63% to 66% with no accuracy loss. Additional exploration of the precision space may yield even shorter precisions without sacrificing accuracy. Modest additional improvements are possible with a loss of 1% in accuracy.
Execution Time: Table 2 reports TRT’s performance and energy efficiency relative to DaDN for the precision profiles in Table 1 separately for the fully-connected layers, for the convolutional layers,
99% Accuracy
and the whole network. For the 100% profile, where no accuracy is lost, TRT yields, on average, a speedup of 1.61× over DaDN on FCLs. With the 99% profile, it improves to 1.73×. There are two main reasons the ideal speedup can’t be reached in practice: dispatch overhead and underutilization. Dispatch overhead occurs on the initial PLw cycles of execution, where the serial weight loading process prevents any useful products to be performed. In practice, this overhead is less than 2% for any given network, although it can be as high as 6% for the smallest layers. Underutilization can happen when the number of output neurons is not a power of two, or lower than 256. The last classifier layers of networks designed towards recognition of ImageNet (Russakovsky et al. (2014)) categories all provide 1000 output neurons, which leads to 2.3% of the SIPs being idle.
We have also performed an evaluation of NeuralTalk LSTM Karpathy & Li (2014) which uses long short-term memory to automatically generate image captions. Precision can be reduced down to 11 bits withouth affecting the accuracy of the predictions (measured as the BLEU score when compared to the ground truth) resulting in a ideal performance improvement of 1.45× translating into a 1.38× speedup with TRT .
Energy Efficiency: This section compares the energy efficiency or simply efficiency of TRT and DaDN. Energy Efficiency is the inverse of the relative energy consumption of the two designs. The average efficiency improvement with TRT across all networks and layers for the 100% profile is 1.17×. In the FCLs, TRT is not as efficient as DaDN, however, the energy efficiency for CVLs more than compensates when whole networks are considered except for VGG 19. Regardless, performance would not scale linearly if DaDN was to include more tiles in an attempt to match TRT’s performance: under-utilization for most layers in these networks would severely reduce any performance improvements delivered via additional tiles under DaDN. Overall, efficiency primarily comes from the reduction in effective computation following the use of reduced precision arithmetic for the inner product operations. Furthermore, the amount of data that has to be transmitted from the SB and the traffic between the central eDRAM and the SIPs is decreased proportionally to the chosen
precision. When the per layer precisions are reduced adequately TRT becomes more efficient than DaDN.
Area Overhead: Table 3 reports the area breakdown of TRT and DaDN. Over the full chip, TRT needs 1.49× the area compared to DaDN while delivering on average a 1.90× improvement in speed. Generally, performance would scale sublinearly with area for DaDN due to underutilization. The 2-bit variant, which has a lower area overhead, is described in detail in the next section.
4.3 TWO-BIT AT ONCE PERFORMANCE EVALUATION
We evaluate the performance for a multi-bit design as described in section 3.4, where 2 bits are processed every cycle in as half as many total SIPs. The precisions used are the same as indicated in Table 1 for 100% accuracy, rounded up to the next multiple of two. The results are shown in Table 4. The 2-bit TRT always improves performance compared to DaDN as the “vs. DaDN” columns show. Compared to the 1-bit TRT performance is slightly lower however given that the area of the 2-bit TRT is much lower, this can be a good trade-off. Overall, there are two forces at work that shape performance relative to the 1-bit TRT . There is performance potential lost due to rounding all precisions to an even number, and there is performance benefit by requiring less parallelism. The time needed to serially load the first bundle of weights is also reduced. In VGG 19 the performance benefit due to the lower parallelism requirement outweights the performance loss due to precision rounding. In all other cases, the reverse is true.
A hardware synthesis and layout of both DaDN and TRT’s 2-bit variant using TSMC 65nm typical case libraries shows that the total area overhead can be as low as 24.9%, with an improved energy efficiency in fully connected layers of 1.24× on average.
5 RELATED WORK AND LIMITATIONS OF THIS WORK
The recent success of Deep Learning has led to several proposals for hardware acceleration of DNNs. This section reviews some of these recent efforts. However, specialized hardware designs for neural networks is a field with a relatively long history. Relevant to TRT , bit-serial processing hardware for neural networks has been proposed several decades ago, e.g., Svensson & Nordstrom (1990); Murray et al. (1988). While the performance of these designs scales with precision it would be lower than that of an equivalently configured bit-parallel engine. For example, Svensson & Nordstrom (1990) uses an interesting bit-serial multiplier which requires O(4 × p) cycles, where p the precision in bits. Furthermore, as semiconductor technology has progressed the number of resources that can be
put on chip and the trade offs (e.g., relative speed of memory vs. transistors vs. wires) are today vastly different facilitating different designs. However, truly bit-serial processing such as that used in the aforementioned proposals needs to be revisited with today’s technology constraints due to its potentially high compute density (compute bandwidth delivered per area).
In general, hardware acceleration for DNNs has recently progressed in two directions: 1) considering more general purpose accelerators that can support additional machine learing algorithms, and 2) considering further improvements primarily for convolutional neural networks and the two most dominant in terms of execution time layer types: convolutional and fully-connected. In the first category there are accelerators such as Cambricon Liu et al. (2016) and Cambricon-X Zhang et al. (2016). While targeting support for more machine learning algorithms is desirable, work on further optimizing performance for specific algorithms such as TRT is valuable and needs to be pursued as it will affect such general purpose accelerators.
TRT is closely related to Stripes Judd et al. (2016c;a) whose execution time scales with precision but only for CVLs. STR does not improve performance for FCLs. TRT improves upon STR by enabling: 1) performance improvements for FCLs, and 2) slicing the activation computation across multiple SIPs thus preventing underutilization for layers with fewer than 4K outputs. Pragmatic uses a similar in spirit organization to STR but its performance on CVLs depends only on the number of activation bits that are 1 Albericio et al. (2016b). It should be possible to apply the TRT extensions to Pragmatic, however, performance in FCLs will still be dictated by weight precision. The area and energy overheads would need to be amortized by a commensurate performance improvement.
The Efficient Inference Engine (EIE) uses synapse pruning, weight compression, zero activation elimination, and network retraining to drastically reduce the amount of computation and data communication when processing fully-connected layers Han et al. (2016). An appropriately configured EIE will outperform TRT for FCLs, provided that the network is pruned and retrained. However, the two approaches attack a different component of FCL processing and there should be synergy between them. Specifically, EIE currently does not exploit the per layer precision variability of DNNs and relies on retraining the network. It would be interesting to study how EIE would benefit from a TRT-like compute engine where EIE’s data compression and pruning is used to create vectors of weights and activations to be processed in parallel. EIE uses single-lane units whereas TRT uses a coarser-grain lane arrangement and thus would be prone to more imbalance. A middle ground may be able to offer some performance improvement while compensating for cross-lane imbalance.
Eyeriss uses a systolic array like organization and gates off computations for zero activations Chen, Yu-Hsin and Krishna, Tushar and Emer, Joel and Sze, Vivienne (2016) and targets primarily highenergy efficiency. An actual prototype has been built and is in full operation. Cnvlutin is a SIMD accelerator that skips on-the-fly ineffectual activations such as those that are zero or close to zero Albericio et al. (2016a). Minerva is a DNN hardware generator which also takes advantage of zero activations and that targets high-energy efficiency Reagen et al. (2016). Layer fusion can further reduce off-chip communication and create additional parallelism Alwani et al. (2016). As multiple layers are processed concurrently, a straightforward combination with TRT would use the maximum of the precisions when layers are fused.
Google’s Tensor Processing Unit uses quantization to represent values using 8 bits Jouppi (2016) to support TensorFlow Abadi et al. (2015). As Table 1 shows, some layers can use lower than 8 bits of precision which suggests that even with quantization it may be possible to use fewer levels and to potentially benefit from an engine such as TRT .
Limitations: As in DaDN this work assumed that each layer fits on-chip. However, as networks evolve it is likely that they will increase in size thus requiring multiple TRT nodes as was suggested in DaDN. However, some newer networks tend to use more but smaller layers. Regardless, it would be desirable to reduce the area cost of TRT most of which is due to the eDRAM buffers. We have not explored this possibility in this work. Proteus Judd et al. (2016b) is directly compatible with TRT and can reduce memory footprint by about 60% for both convolutional and fully-connected layers. Ideally, compression, quantization and pruning similar in spirit to EIE Han et al. (2016) would be used to reduce computation, communication and footprint. General memory compresion Mittal & Vetter (2016) techniques offer additional opportunities for reducing footprint and communication.
We evaluated TRT only on CNNs for image classification. Other network architectures are important and the layer configurations and their relative importance varies. TRT enables performance
improvements for two of the most dominant layer types. We have also provided some preliminary evidence that TRT works well for NeuralTalk LSTM Karpathy & Li (2014). Moreover, by enabling output activation computation slicing it can accommodate relatively small layers as well.
Applying some of the concepts that underlie the TRT design to other more general purpose accelerators such as Cambricon Liu et al. (2016) or graphics processors would certainly be more preferable than a dedicated accelerator in most application scenarios. However, these techniques are best first investigated into specific designs and then can be generalized appropriately.
We have evaluated TRT only for inference only. Using an engine whose performance scales with precision would provide another degree of freedom for network training as well. However, TRT needs to be modified accordingly to support all the operations necessary during training and the training algorithms need to be modified to take advantage of precision adjustments.
This section commented only on related work on digital hardware accelerators for DNNs. Advances at the algorithmic level would impact TRT as well or may even render it obsolete. For example, work on using binary weights Courbariaux et al. (2015) would obviate the need for an accelerator whose performance scales with weight precision. Investigating TRT’s interaction with other network types and architectures and other machine learning algorithms is left for future work.
6 CONCLUSION
This work presented Tartan an accelerator for inference with Deep Learning Networks whose performance scales inversely linearly with the number of bits used to represent values in fully-connected and convolutional layers. TRT also enables on-the-fly accuracy vs. performance and energy efficiency trade offs and its benefits were demonstrated over a set of popular image classification networks. The new key ideas in TRT are: 1) Supporting both the bit-parallel and the bit-serial loading of weights into processing units to facilitate the processing of either convolutional or fullyconnected layers, and 2) cascading the adder trees of various subunits (SIPs) to enable slicing the output computation thus reducing or eliminating cross-lane imbalance for relatively small layers.
TRT opens up a new direction for research in inference and training by enabling precision adjustments to translate into performance and energy savings. These precisions adjustments can be done statically prior to execution or dynamically during execution. While we demonstrated TRT for inference only, we believe that TRT , especially if combined with Pragmatic, opens up a new direction for research in training as well. For systems level research and development, TRT with its ability to trade off accuracy for performance and energy efficiency enables a new degree of adaptivity for operating systems and applications.

Summary:
The paper describes how the DaDianNao (DaDN) DNN accelerator can be improved by employing bit serial arithmetic.  They replace the bit-parallel multipliers in DaDN with multipliers that accept the weights in parallel but the activations serially (serial x parallel multipliers).  They increase the number of units keeping the total number of adders constant.  This enables them to tailor the time and energy consumed to the number of bits used to represent activations.  They show how their configuration can be used to process both fully-connected and convolutional layers of DNNs.
Strengths:
Using variable precision for each layer of the network is useful - but was previously reported in Judd (2015)
Good evaluation including synthesis - but not place and route - of the units.  Also this evaluation is identical to that in Judd (2016b)
Weaknesses:
The idea of combining bit-serial arithmetic with the DaDN architecture is a small one.
The authors have already published almost everything that is in this paper at Micro 2016 in Judd (2016b).  The increment here is the analysis of the architecture on fully-connected layers.  Everything else is in the previous publication.
The energy gains are small - because the additional flip-flop energy of shifting the activations in almost offsets the energy saved on reducing the precision of the arithmetic.
The authors don’t compare to more conventional approaches to variable precision - using bit-parallel arithmetic units but data gating the LSBs so that only the relevant portion of the arithmetic units toggle.  This would not provide any speedup, but would likely provide better energy gains than the bit-serial x bit-parallel approach.
Overall:
The Tartan and Stripes architectures are interesting but the incremental contribution of this paper (adding support for fully-connected layers) over the three previous publications on this topic, and in particular Judd (2016b) is very small.  This idea is worth one good paper, not four.


HIGHER ORDER RECURRENT NEURAL NETWORKS
In this paper, we study novel neural network structures to better model long term dependency in sequential data. We propose to use more memory units to keep track of more preceding states in recurrent neural networks (RNNs), which are all recurrently fed to the hidden layers as feedback through different weighted paths. By extending the popular recurrent structure in RNNs, we provide the models with better short-term memory mechanism to learn long term dependency in sequences. Analogous to digital filters in signal processing, we call these structures as higher order RNNs (HORNNs). Similar to RNNs, HORNNs can also be learned using the back-propagation through time method. HORNNs are generally applicable to a variety of sequence modelling tasks. In this work, we have examined HORNNs for the language modeling task using two popular data sets, namely the Penn Treebank (PTB) and English text8. Experimental results have shown that the proposed HORNNs yield the state-of-the-art performance on both data sets, significantly outperforming the regular RNNs as well as the popular LSTMs.
1 INTRODUCTION
In the recent resurgence of neural networks in deep learning, deep neural networks have achieved successes in various real-world applications, such as speech recognition, computer vision and natural language processing. Deep neural networks (DNNs) with a deep architecture of multiple nonlinear layers are an expressive model that can learn complex features and patterns in data. Each layer of DNNs learns a representation and transfers them to the next layer and the next layer may continue to extract more complicated features, and finally the last layer generates the desirable output. From early theoretical work, it is well known that neural networks may be used as the universal approximators to map from any fixed-size input to another fixed-size output. Recently, more and more empirical results have demonstrated that the deep structure in DNNs is not just powerful in theory but also can be reliably learned in practice from a large amount of training data.
Sequential modeling is a challenging problem in machine learning, which has been extensively studied in the past. Recently, many deep neural network based models have been successful in this area, as shown in various tasks such as language modeling Mikolov (2012), sequence generation Graves (2013); Sutskever et al. (2011), machine translation Sutskever et al. (2014) and speech recognition Graves et al. (2013). Among various neural network models, recurrent neural networks (RNNs) are appealing for modeling sequential data because they can capture long term dependency in sequential data using a simple mechanism of recurrent feedback. RNNs can learn to model sequential data over an extended period of time, then carry out rather complicated transformations on the sequential data. RNNs have been theoretically proved to be a turing complete machine Siegelmann & Sontag (1995). RNNs in principle can learn to map from one variable-length sequence to another. When unfolded in time, RNNs are equivalent to very deep neural networks that share model parameters and receive the input at each time step. The recursion in the hidden layer of RNNs can act as an excellent memory mechanism for the networks. In each time step, the learned recursion weights may decide what information to discard and what information to keep in order to relay onwards along time. While RNNs are theoretically powerful, the learning of RNNs needs to use the back-propagation through time (BPTT) method Werbos (1990) due to the internal recurrent cycles. Unfortunately, in practice, it turns out to be rather difficult to train RNNs to capture long-term dependency due to the fact that
the gradients in BPTT tend to either vanish or explode Bengio et al. (1994). Many heuristic methods have been proposed to solve these problems. For example, a simple method, called gradient clipping, is used to avoid gradient explosion Mikolov (2012). However, RNNs still suffer from the vanishing gradient problem since the gradients decay gradually as they are back-propagated through time. As a result, some new recurrent structures are proposed, such as long short-term memory (LSTM) Hochreiter & Schmidhuber (1997) and gated recurrent unit (GRU) Cho et al. (2014). These models use some learnable gates to implement rather complicated feedback structures, which ensure that some feedback paths can allow the gradients to flow back in time effectively. These models have given promising results in many practical applications, such as sequence modeling Graves (2013), language modeling Sundermeyer et al. (2012), hand-written character recognition Liwicki et al. (2012), machine translation Cho et al. (2014), speech recognition Graves et al. (2013).
In this paper, we explore an alternative method to learn recurrent neural networks (RNNs) to model long term dependency in sequential data. We propose to use more memory units to keep track of more preceding RNN states, which are all recurrently fed to the hidden layers as feedback through different weighted paths. Analogous to digital filters in signal processing, we call these new recurrent structures as higher order recurrent neural networks (HORNNs). At each time step, the proposed HORNNs directly combine multiple preceding hidden states from various history time steps, weighted by different matrices, to generate the feedback signal to each hidden layer. By aggregating more history information of the RNN states, HORNNs are provided with better short-term memory mechanism than the regular RNNs. Moreover, those direct connections to more previous RNN states allow the gradients to flow back smoothly in the BPTT learning stage. All of these ensure that HORNNs can be more effectively learned to capture long term dependency. Similar to RNNs and LSTMs, the proposed HORNNs are general enough for variety of sequential modeling tasks. In this work, we have evaluated HORNNs for the language modeling task on two popular data sets, namely the Penn Treebank (PTB) and English text8 sets. Experimental results have shown that HORNNs yield the state-of-the-art performance on both data sets, significantly outperforming the regular RNNs as well as the popular LSTMs.
2 RELATED WORK
Hierarchical recurrent neural network proposed in Hihi & Bengio (1996) is one of the earliest papers that attempt to improve RNNs to capture long term dependency in a better way. It proposes to add linear time delayed connections to RNNs to improve the gradient descent learning algorithm to find a better solution, eventually solving the gradient vanishing problem. However, in this early work, the idea of multi-resolution recurrent architectures has only been preliminarily examined for some simple small-scale tasks. This work is somehow relevant to our work in this paper but the higher order RNNs proposed here differs in several aspects. Firstly, we propose to use weighted connections in the structure, instead of simple multi-resolution short-cut paths. This makes our models fall into the category of higher order models. Secondly, we have proposed to use various pooling functions in generating the feedback signals, which is critical in normalizing the dynamic ranges of gradients flowing from various paths. Our experiments have shown that the success of our models is largely attributed to this technique.
The most successful approach to deal with vanishing gradients so far is to use long short term memory (LSTM) model Hochreiter & Schmidhuber (1997). LSTM relies on a fairly sophisticated structure made of gates to control flow of information to the hidden neurons. The drawback of the LSTM is that it is complicated and slow to learn. The complexity of this model makes the learning very time consuming, and hard to scale for larger tasks. Another approach to address this issue is to add a hidden layer to RNNs Mikolov et al. (2014). This layer is responsible for capturing longer term dependencies in input data by making its weight matrix close to identity. Recently, clockwork RNNs Koutnik et al. (2014) are proposed to address this problem as well, which splits each hidden layer into several modules running at different clocks. Each module receives signals from input and computes its output at a predefined clock rate. Gated feedback recurrent neural networks Chung et al. (2015) attempt to implement a generalized version using the gated feedback connection between layers of stacked RNNs, allowing the model to adaptively adjust the connection between consecutive hidden layers.
Besides, short-cut skipping connections were considered earlier in Wermter (1992), and more recently have been found useful in learning very deep feed-forward neural networks as well, such as Lee et al. (2014); He et al. (2015). These skipping connections between various layers of neural networks can improve the flow of information in both forward and backward passes. Among them, highway networks Srivastava et al. (2015) introduce rather sophisticated skipping connections between layers, controlled by some gated functions.
3 HIGHER ORDER RECURRENT NEURAL NETWORKS
A recurrent neural network (RNN) is a type of neural network suitable for modeling a sequence of arbitrary length. At each time step t, an RNN receives an input xt, the state of the RNN is updated recursively as follows (as shown in the left part of Figure 1):
ht = f(Winxt +Whht−1) (1)
where f(·) is an nonlinear activation function, such as sigmoid or rectified linear (ReLU), and Win is the weight matrix in the input layer and Wh is the state to state recurrent weight matrix. Due to the recursion, this hidden layer may act as a short-term memory of all previous input data.
Given the state of the RNN, i.e., the current activation signals in the hidden layer ht, the RNN generates the output according to the following equation:
yt = g(Woutht) (2)
where g(·) denotes the softmax function and Wout is the weight matrix in the output layer. In principle, this model can be trained using the back-propagation through time (BPTT) algorithm Werbos (1990). This model has been used widely in sequence modeling tasks like language modeling Mikolov (2012).
3.1 HIGHER ORDER RNNS (HORNNS)
RNNs are very deep in time and the hidden layer at each time step represents the entire input history, which acts as a short-term memory mechanism. However, due to the gradient vanishing problem in back-propagation, it turns out to be very difficult to learn RNNs to model long-term dependency in sequential data.
In this paper, we extend the standard RNN structure to better model long-term dependency in sequential data. As shown in the right part of Figure 1, instead of using only the previous RNN state as the feedback signal, we propose to employ multiple memory units to generate the feedback signal at each time step by directly combining multiple preceding RNN states in the past, where these timedelayed RNN states go through separate feedback paths with different weight matrices. Analogous to the filter structures used in signal processing, we call this new recurrent structure as higher order RNNs, HORNNs in short. The order of HORNNs depends on the number of memory units used for feedback. For example, the model used in the right of Figure 1 is a 3rd-order HORNN. On the other hand, regular RNNs may be viewed as 1st-order HORNNs.
In HORNNs, the feedback signal is generated by combining multiple preceding RNN states. Therefore, the state of an N -th order HORNN is recursively updated as follows:
ht = f ( Winxt +
N∑ n=1 Whnht−n
) (3)
where {Whn | n = 1, · · ·N} denotes the weight matrices used for various feedback paths. Similar to
RNNs, HORNNs can also be unfolded in time to get rid of the recurrent cycles. As shown in Figure 2, we unfold a 3rd-order HORNN in time, which clearly shows that each HORNN state is explicitly decided by the current input xt and all previous 3 states in the past. This structure looks similar to the skipping short-cut paths in deep neural networks but each path in HORNNs maintains a learnable weight matrix. The new structure in HORNNs can significantly improve the model capacity to capture long-term dependency in sequential data. At each time step, by explicitly aggregating multiple preceding hidden activities, HORNNs may derive a good representation of the history information in sequences, leading to a significantly enhanced short-term memory mechanism.
During the backprop learning procedure, these skipping paths directly connected to more previous hidden states of HORNNs may allow the gradients to flow more easily back in time, which eventually leads to a more effective learning of models to capture long term dependency in sequences. Therefore, this structure may help to largely alleviate the notorious problem of vanishing gradients in the RNN learning.
Obviously, HORNNs can be learned using the same BPTT algorithm as regular RNNs, except that the error signals at each time step need to be back-propagated to multiple feedback paths in the network. As shown in Figure 3, for a 3rd-order HORNN, at each time step t, the error signal from the hidden layer ht will have to be back-propagated into four different paths: i) the first one back to the input layer, xt; ii) three more feedback paths leading to three different histories in time scales, namely ht−1, ht−2 and ht−3.
Interestingly enough, if we use a fully-unfolded implementation for HORNNs as in Figure 2, the overall computation complexity is comparable with regular RNNs. Given a whole sequence, we may first simultaneously compute all hidden activities (from xt to ht for all t). Secondly, we recursively update ht for all t using eq.(3). Finally, we use GPUs to compute all outputs together from the updated hidden states (from ht to yt for all t) based on eq.(2). The backward pass in learning can also be implemented in the same three-step procedure. Except the recursive updates in the second step (this issue remains the same in regular RNNs), all remaining computation steps can be formulated as large matrix multiplications. As a result, the computation of HORNNs can be implemented fairly efficiently using GPUs.
3.2 POOLING FUNCTIONS FOR HORNNS
As discussed above, the shortcut paths in HORNNs may help the models to capture long-term dependency in sequential data. On the other hand, they may also complicate the learning in a different way. Due to different numbers of hidden layers along various paths, the signals flowing from different paths may vary dramatically in the dynamic range. For example, in the forward pass in Figure 2, three different feedback signals from different time scales, e.g. ht−1, ht−2 and ht−3, flow into
the hidden layer to compute the new hidden state ht. The dynamic range of these signals may vary dramatically from case to case. The situation may get even worse in the backward pass during the BPTT learning. For example, in a 3rd-order HORNN in Figure 2, the node ht−3 may directly receive an error signal from the node ht. In some cases, it may get so strong as to overshadow other error signals coming from closer neighbours of ht−1 and ht−2. This may impede the learning of HORNNs, yielding slow convergence or even poor performance.
Here, we have proposed to use some pooling functions to calibrate the signals from different feedback paths before they are used to recursively generate a new hidden state, as shown in Figure 4. In the following, we will investigate three different choices for the pooling function in Figure 4, including max-based pooling, FOFE-based pooling and gated pooling.
3.2.1 MAX-BASED POOLING
Max-based pooling is a simple strategy that chooses the most responsive unit (exhibiting the largest activation value) among various paths to transfer to the hidden layer to generate the new hidden state. Many biological experiments have shown that biological neuron networks tend to use a similar strategy in learning and firing.
In this case, instead of using eq.(3), we use the following formula to update the hidden state of HORNNs: ht = f ( Winxt +max N n=1 (Whnht−n) ) (4)
where maximization is performed element-wisely to choose the maximum value in each dimension to feed to the hidden layer to generate the new hidden state. The aim here is to capture the most relevant feature and map it to a fixed predefined size.
The max pooling function is simple and biologically inspired. However, the max pooling strategy also has some serious disadvantages. For example, it has no forgetting mechanism and the signals may get stronger and stronger. Furthermore, it loses the order information of the preceding histories since it only choose the maximum values but it does not know where the maximum comes from.
3.2.2 FOFE-BASED POOLING
The fixed-size ordinally-forgetting encoding (FOFE) method was proposed in Zhang et al. (2015) to encode any variable-length sequence of data into a fixed-size representation. In FOFE, a single forgetting factor α (0 < α < 1) is used to encode the position information in sequences based on the idea of exponential forgetting to derive invertible fixed-size representations. In this work, we borrow this simple idea of exponential forgetting to calibrate all preceding histories using a pre-selected forgetting factor as follows:
ht = f ( Winxt +
N∑ n=1 αn ·Whnht−n
) (5)
where the forgetting factor α is manually pre-selected between 0 < α < 1. The above constant coefficients related to α play an important role in calibrating signals from different paths in both
forward and backward passes of HORNNs since they slightly underweight the older history over the recent one in an explicit way.
3.2.3 GATED HORNNS
In this section, we follow the ideas of the learnable gates in LSTMs Hochreiter & Schmidhuber (1997) and GRUs Cho et al. (2014) as well as the recent soft-attention in Bahdanau et al. (2014). Instead of using constant coefficients derived from a forgetting factor, we may let the network automatically determine the combination weights based on the current state and input. In this case, we may use sigmoid gates to compute combination weights to regulate the information flowing from various feedback paths. The sigmoid gates take the current data and previous hidden state as input to decide how to weight all of the precede hidden states. The gate function weights how the current hidden state is generated based on all the previous time-steps of the hidden layer. This allows the network to potentially remember information for a longer period of time. In a gated HORNN, the hidden state is recursively computed as follows:
ht = f ( Winxt +
N∑ n=1 rn ( Whnht−n )) (6)
where denotes element-wise multiplication of two equally-sized vectors, and the gate signal rn is calculated as
rn = σ (W g 1nxt +W g 2nht−n) (7)
where σ(·) is the sigmoid function, and W g1n and W g 2n denote two weight matrices introduced for each gate.
Note that the computation complexity of gated HORNNs is comparable with LSTMs and GRUs, significantly exceeding the other HORNN structures because of the overhead from the gate functions in eq. (7).
4 EXPERIMENTS
In this section, we evaluate the proposed higher order RNNs (HORNNs) on several language modeling tasks. A statistical language model (LM) is a probability distribution over sequences of words in natural languages. Recently, neural networks have been successfully applied to language modeling Bengio et al. (2003); Mikolov et al. (2011), yielding the state-of-the-art performance. In language modeling tasks, it is quite important to take advantage of the long-term dependency of natural languages. Therefore, it is widely reported that RNN based LMs can outperform feedforward neural networks in language modeling tasks. We have chosen two popular LM data sets, namely the Penn Treebank (PTB) and English text8 sets, to compare our proposed HORNNs with traditional n-gram LMs, RNN-based LMs and the state-of-the-art performance obtained by LSTMs Graves (2013); Mikolov et al. (2014), FOFE based feedforward NNs Zhang et al. (2015) and memory networks Sukhbaatar et al. (2015).
In our experiments, we use the mini-batch stochastic gradient decent (SGD) algorithm to train all neural networks. The number of back-propagation through time (BPTT) steps is set to 30 for all recurrent models. Each model update is conducted using a mini-batch of 20 subsequences, each of which is of 30 in length. All model parameters (weight matrices in all layers) are randomly initialized based on a Gaussian distribution with zero mean and standard deviation of 0.1. A hard clipping is set to 5.0 to avoid gradient explosion during the BPTT learning. The initial learning rate is set to 0.5 and we halve the learning rate at the end of each epoch if the cross entropy function on the validation set does not decrease. We have used the weight decay, momentum and column normalization Pachitariu & Sahani (2013) in our experiments to improve model generalization. In the FOFE-based pooling function for HORNNs, we set the forgetting factor, α, to 0.6. We have used 400 nodes in each hidden layer for the PTB data set and 500 nodes per hidden layer for the English text8 set. In our experiments, we do not use the dropout regularization Zaremba et al. (2014) in all experiments since it significantly slows down the training speed, not applicable to any larger corpora. 1
1We will soon release the code for readers to reproduce all results reported in this paper.
4.1 LANGUAGE MODELING ON PTB
The standard Penn Treebank (PTB) corpus consists of about 1M words. The vocabulary size is limited to 10k. The preprocessing method and the way to split data into training/validation/test sets are the same as Mikolov et al. (2011). PTB is a relatively small text corpus. We first investigate various model configurations for the HORNNs based on PTB and then compare the best performance with other results reported on this task.
4.1.1 EFFECT OF ORDERS IN HORNNS
In the first experiment, we first investigate how the used orders in HORNNs may affect the performance of language models (as measured by perplexity). We have examined all different higher order model structures proposed in this paper, including HORNNs and various pooling functions in HORNNs. The orders of these examined models varies among 2, 3 and 4. We have listed the performance of different models on PTB in Table 1. As we may see, we are able to achieve a significant improvement in perplexity when using higher order RNNs for language models on PTB, roughly 10-20 reduction in PPL over regular RNNs. We can see that performance may improve slightly when the order is increased from 2 to 3 but no significant gain is observed when the order is further increased to 4. As a result, we choose the 3rd-order HORNN structure for the following experiments. Among all different HORNN structures, we can see that FOFE-based pooling and gated structures yield the best performance on PTB.
In language modeling, both input and output layers account for the major portion of model parameters. Therefore, we do not significantly increase model size when we go to higher order structures. For example, in Table 1, a regular RNN contains about 8.3 millions of weights while a 3rd-order HORNN (the same for max or FOFE pooling structures) has about 8.6 millions of weights. In comparison, an LSTM model has about 9.3 millions of weights and a 3rd-order gated HORNN has about 9.6 millions of weights.
As for the training speed, most HORNN models are only slightly slower than regular RNNs. For example, one epoch of training on PTB running in one NVIDIA’s TITAN X GPU takes about 80 seconds for an RNN, about 120 seconds for a 3rd-order HORNN (the same for max or FOFE pooling structures). Similarly, training of gated HORNNs is also slightly slower than LSTMs. For example, one epoch on PTB takes about 200 seconds for an LSTM, and about 225 seconds for a 3rd-order gates HORNN.
4.1.2 MODEL COMPARISON ON PENN TREEBANK
At last, we report the best performance of various HORNNs on the PTB test set in Table 2. We compare our 3rd-order HORNNs with all other models reported on this task, including RNN Mikolov et al. (2011), stack RNN Pascanu et al. (2014), deep RNN Pascanu et al. (2014), FOFE-FNN Zhang et al. (2015) and LSTM Graves (2013). 2 From the results in Table 2, we can see that our proposed higher order RNN architectures significantly outperform all other baseline models reported on this task. Both FOFE-based pooling and gated HORNNs have achieved the state-of-the-art performance,
2All models in Table 2 do not use the dropout regularization, which is somehow equivalent to data augmentation. In Zaremba et al. (2014); Kim et al. (2015), the proposed LSTM-LMs (word level or character level) achieve lower perplexity but they both use the dropout regularization and much bigger models and it takes days to train the models, which is not applicable to other larger tasks.
Table 2: Perplexities on the PTB test set for various examined models.
Models Test KN 5-gram Mikolov et al. (2011) 141 RNN Mikolov et al. (2011) 123 CSLM5Aransa et al. (2015) 118.08 LSTM Graves (2013) 117 genCNN Wang et al. (2015) 116.4 Gated word&charMiyamoto & Cho (2016) 113.52 E2E Mem Net Sukhbaatar et al. (2015) 111 Stack RNN Pascanu et al. (2014) 110 Deep RNN Pascanu et al. (2014) 107 FOFE-FNN Zhang et al. (2015) 108 HORNN (3rd order) 108 Max HORNN (3rd order) 109 FOFE HORNN (3rd order) 101 Gated HORNN (3rd order) 100
Table 3: Perplexities on the text8 test set for various models.
Models Test RNN Mikolov et al. (2014) 184 LSTM Mikolov et al. (2014) 156 SCRNN Mikolov et al. (2014) 161 E2E Mem Net Sukhbaatar et al. (2015) 147 HORNN (3rd order) 172 Max HORNN (3rd order) 163 FOFE HORNN (3rd order) 154 Gated HORNN (3rd order) 144
i.e., 100 in perplexity on this task. To the best of our knowledge, this is the best reported performance on PTB under the same training condition.
4.2 LANGUAGE MODELING ON ENGLISH TEXT8
In this experiment, we will evaluate our proposed HORNNs on a much larger text corpus, namely the English text8 data set. The text8 data set contains a preprocessed version of the first 100 million characters downloaded from the Wikipedia website. We have used the same preprocessing method as Mikolov et al. (2014) to process the data set to generate the training and test sets. We have limited the vocabulary size to about 44k by replacing all words occurring less than 10 times in the training set with an <UNK> token. The text8 set is about 20 times larger than PTB in corpus size. The model training on text8 takes longer to finish. We have not tuned hyperparameters in this data set. We simply follow the best setting used in PTB to train all HORNNs for the text8 data set. Meanwhile, we also follow the same learning schedule used in Mikolov et al. (2014): We first initialize the learning rate to 0.5 and run 5 epochs using this learning rate; After that, the learning rate is halved at the end of every epoch.
Because the training is time-consuming, we have only evaluated 3rd-order HORNNs on the text8 data set. The perplexities of various HORNNs are summarized in Table 3. We have compared our HORNNs with all other baseline models reported on this task, including RNN Mikolov et al. (2014), LSTM Mikolov et al. (2014), SCRNN Mikolov et al. (2014) and end-to-end memory networks Sukhbaatar et al. (2015). Results have shown that all HORNN models work pretty well in this data set except the normal HORNN significantly underperforms the other three models. Among them, the gated HORNN model has achieved the best performance, i.e., 144 in perplexity on this task, which is slightly better than the recent result obtained by end-to-end memory networks (using a rather complicated structure). To the best of our knowledge, this is the best performance reported on this task.
5 CONCLUSIONS
In this paper, we have proposed some new structures for recurrent neural networks, called as higher order RNNs (HORNNs). In these structures, we use more memory units to keep track of more preceding RNN states, which are all fed along various feedback paths to the hidden layer to generate the feedback signals. In this way, we may enhance the model to capture long term dependency in sequential data. Moreover, we have proposed to use several types of pooling functions to calibrate multiple feedback paths. Experiments have shown that the pooling technique plays a critical role in learning higher order RNNs effectively. In this work, we have examined HORNNs for the language modeling task using two popular data sets, namely the Penn Treebank (PTB) and text8 sets. Experimental results have shown that the proposed higher order RNNs yield the state-of-the-art per-
formance on both data sets, significantly outperforming the regular RNNs as well as the popular LSTMs. As the future work, we are going to continue to explore HORNNs for other sequential modeling tasks, such as speech recognition, sequence-to-sequence modelling and so on.

This paper proposes an idea of looking n-steps backward when modelling sequences with RNNs. The proposed RNN does not only use the previous hidden state (t-1) but also looks further back ( (t - k) steps, where k=1,2,3,4 ). The paper also proposes a few different ways to aggregate multiple hidden states from the past.
The reviewer can see few issues with this paper.
Firstly, the writing of this paper requires improvement. The introduction and abstract are wasting too much space just to explain unrelated facts or to describe already well-known things in the literature. Some of the statements written in the paper are misleading. For instance, it explains, “Among various neural network models, recurrent neural networks (RNNs) are appealing for modeling sequential data because they can capture long term dependency in sequential data using a simple mechanism of recurrent feedback” and then it says RNNs cannot actually capture long-term dependencies that well. RNNs are appealing in the first place because they can handle variable length sequences and can model temporal relationships between each symbol in a sequence. The criticism against LSTMs is hard to accept when it says: LSTMs are slow and because of the slowness, they are hard to scale at larger tasks. But we all know that some companies are already using gigantic seq2seq models for their production (LSTMs are used as building blocks in their systems). This indicates that the LSTMs can be practically used in a very large-scale setting.
Secondly, the idea proposed in the paper is incremental and not new to the field. There are other previous works that propose to use direct connections to the previous hidden states [1]. However, the previous works do not use aggregation of multiple number of previous hidden states. Most importantly, the paper fails to deliver a proper analysis on whether its main contribution is actually helpful to improve the problem posed in the paper. The new architecture is said that it handles the long-term dependencies better, however, there is no rigorous proof or intuitive design in the architecture that help us to understand why it should work better. By the design of the architecture, and speaking in very high-level, it seems like the model maybe helpful to mitigate the vanishing gradients issue by a linear factor. It is always a good practice to have at least one page to analyze the empirical findings in the paper.
Thirdly, the baseline models used in this paper are very weak. Their are plenty of other models that are trained and tested on word-level language modelling task using Penn Treebank corpus, but the paper only contains a few of outdated models. I cannot fully agree on the statement “To the best of our knowledge, this is the best performance on PTB under the same training condition”, these days, RNN-based methods usually score below 80 in terms of the test perplexity, which are far lower than 100 achieved in this paper.
[1] Zhang et al., “Architectural Complexity Measures of Recurrent Neural Networks”, NIPS’16