Flexpoint: An Adaptive Numerical Format for Efficient Training of Deep Neural Networks
Deep neural networks are commonly developed and trained in 32-bit floating point format. Significant gains in performance and energy efficiency could be realized by training and inference in numerical formats optimized for deep learning. Despite advances in limited precision inference in recent years, training of neural networks in low bit-width remains a challenging problem. Here we present the Flexpoint data format, aiming at a complete replacement of 32-bit floating point format training and inference, designed to support modern deep network topologies without modifications. Flexpoint tensors have a shared exponent that is dynamically adjusted to minimize overflows and maximize available dynamic range. We validate Flexpoint by training AlexNet [1], a deep residual network [2, 3] and a generative adversarial network [4], using a simulator implemented with the neon deep learning framework. We demonstrate that 16-bit Flexpoint closely matches 32-bit floating point in training all three models, without any need for tuning of model hyperparameters. Our results suggest Flexpoint as a promising numerical format for future hardware for training and inference.
I find the proposed technique interesting, but I believe that these problem might be better situated for a hardware venue. This seems to be more a contribution on hardware representations, as compared to previous techniques like xor-net which don't ask for a new hardware representation and had in mind an application to compression of networks for phones, and evaluated degradation of final performance.  However, I do think that the paper could be of interest to the NIPS community as similar papers have been published in AI conferences.
The paper is well written and easy to understand.  The upsides of the method and the potential problems are well described and easy to follow. Improvements over previous techniques seem somewhat incremental, and for many applications it's not clear that training with a low precision format is better than training with a high precision format and then cropping down, and no experiments are performed to explore that.
My opinion on the evaluation is mixed.  For one, no experiments are done to compare to previous (crop to low precision after training) techniques.  Second, plotting validation error seems natural, but I would also like to see test numbers.  I notice that the validation result for Alexnet on ILSVRC is about 20% error, while alexnet originally reported 17% error on ILSVRC-11 and 16.4% on ILSVR-2012 test.  Allowing for direct comparison would improve the paper.  The results for CIFAR (~5% error) seem to be near state of the art.  Qualitative GAN results appear to be good.
Further Comments:
After discussion, I believe that the paper could be accepted (increased to a 7 from a 6).  Other reviewers have argued that the authors need to provide evidence that such a system would be implementable in hardware, be faster, and have a lower memory footprint.  I do not believe that those experiments are necessary.  Given the large number of network architectures that people are using currently, which ones should the authors demonstrate on?  If one architecture causes problems, is the paper unacceptable?  I am also somewhat confused that other reviews seem to give no value to drastically reducing the memory footprint (by easy calculation each layer is represented in half as much memory).  Many of the current state of the art network architectures use the maximum GPU memory during training.  While it needs more work, this technique seems to provide a no-loss answer to how to increase network capacity which is very cool indeed
This is also related to my original point about venue, as it is this paper sits somewhere between AI and systems.  If the authors performs further "hardware-centric" experiments it seems the paper would clearly be better suited for a hardware or systems venue.  If we evaluate this work as a network compression paper then the classification accuracy demonstrates success.  A network trained in half as much memory is as accurate as the full memory network.  It deserves consideration by the ACs for acceptance.