Training DNNs with Hybrid Block Floating Point
The wide adoption of DNNs has given birth to unrelenting computing requirements, forcing datacenter operators to adopt domain-specific accelerators to train them. These accelerators typically employ densely packed full-precision floating-point arithmetic to maximize performance per area. Ongoing research efforts seek to further increase that performance density by replacing floating-point with fixedpoint arithmetic. However, a significant roadblock for these attempts has been fixed point's narrow dynamic range, which is insufficient for DNN training convergence. We identify block floating point (BFP) as a promising alternative representation since it exhibits wide dynamic range and enables the majority of DNN operations to be performed with fixed-point logic. Unfortunately, BFP alone introduces several limitations that preclude its direct applicability. In this work, we introduce HBFP, a hybrid BFP-FP approach, which performs all dot products in BFP and other operations in floating point. HBFP delivers the best of both worlds: the high accuracy of floating point at the superior hardware density of fixed point. For a wide variety of models, we show that HBFP matches floating point's accuracy while enabling hardware implementations that deliver up to 8.5× higher throughput.
This work suggests a new approach for fixed+floating point hybrid. To enable this, use a shared exponent per tensor that allows simple conversion between the formats. They use a fixed point representation for GEMMs to allow only mantissa multiplication while using floating point for all other operations. They also introduce a conversion scheme between intermediate representations of weights and activations. The utility of this approach is demonstrated by simulations and hardware implementation.  Overall this is a nice work, with convincing arguments and experiments. Most noteworthy, I enjoyed seeing an empirical justification for favoring a large exponent, while using a trimmed mantissa. This echoes similar observations by others (e.g BF16 format used by Google). I think that additional comparison to Nervana's Flexpoint format [1] is in place, as it also used an exponent shared scheme, although with different choices for exponent and mantissa sizes.  There are some aspects that I will be happy to see covered: 1) Bandwidth vs computational bound operations - your methods will probably not improve run-time for several operations, it's interesting to see some kind of tear-down for a network such as ResNet50.  E.g - where were the bottlenecks for a floating-point implementation, and for yours (see [2])? 2) Batch norm – You mentioned that “While activation traffic is dwarfed by weight traffic in fully connected, in convolutional layers  … not significant throughput factor”. I would argue that batch-norm is highly bandwidth bound, and that it should be discussed further in your work (most noteworthy, as batch-norm is done completely in fp32 from what I gather). 3) Estimating the cost of conversion operations – you mentioned both finding maximum value in tensor and stochastic rounding to be needed for quantization. I’ll be happy to get an estimate for the cost of these operations (runtime) and their importance in terms of accuracy. 4) A more challenging recurrent task – PTB is good, but I’ll be happy to see a more challenging benchmark such as machine translation to be assured that your methods indeed transfer to other models.  Strengths: - Convincing arguments for a novel numerical representation for DNN training - Good empirical experiments  Weaknesses: - Missing analysis for operations that are not computationally bound (batch-norm) - Estimating the relative cost for needed utility operations (maximum finding, stochatic rounding)  [1] - https://arxiv.org/abs/1711.02213 [2] - https://arxiv.org/abs/1709.08145