Shifted and Squeezed 8-bit Floating Point format for Low-Precision Training of Deep Neural Networks
Training with larger number of parameters while keeping fast iterations is an increasingly adopted strategy and trend for developing better performing Deep Neural Network (DNN) models. This necessitates increased memory footprint and computational requirements for training. Here we introduce a novel methodology for training deep neural networks using 8-bit floating point (FP8) numbers. Reduced bit precision allows for a larger effective memory and increased computational speed. We name this method Shifted and Squeezed FP8 (S2FP8). We show that, unlike previous 8-bit precision training methods, the proposed method works out of the box for representative models: ResNet50, Transformer and NCF. The method can maintain model accuracy without requiring fine-tuning loss scaling parameters or keeping certain layers in single precision. We introduce two learnable statistics of the DNN tensors - shifted and squeezed factors that are used to optimally adjust the range of the tensors in 8-bits, thus minimizing the loss in information due to quantization.
The paper suggest the method to train neural networks using 8 bit floating point precision values. The method requires a hardware modification to evaluate per tensor statistics, which intuitively capture dynamic range that a given tensor values may fall into. Those per tensor statistics are used to map 8-bit representation into a rational number. Statistics themselves are calculated using 32 bit floating point numbers. While the method is interesting, I do not think it is practical due to the required hardware modifications. I am by no means not a hardware design expert, but I am not convinced that the gain of using 8 vs 16 bit floating point numbers outweights any extra complexity of hardware implementation. Mapping between representation and actual numerical values is much more complex than when using standard floating point representation (as any given numerical value is defined by a single its 8 bit representation + alpha + beta tensor statistics), integrated circuitry used to execute arithmetic operations would likely be much more complex than when using standard floating point operations. It is plausible that the added complexity (in terms of transistor count) would negate any potential price or energy savings over simply using 16 bit floating point representation. This should definitely be discussed in depth in this paper where the main contribution is an algorithm to be implemented in hardware. As different layers will have different (alpha,beta) tensor statistics, from the computational perspective they would look like different data types which would be cast into the same common type before doing any arithmetic operations on them. This would almost certainly require extra computational steps to be done potentially negating computational benefits over simply using 16 bit floating point numbers. The algorithm evaluates per-tensor and not per-tensor-element statistics. This may work well in cases where all entries in a given tensor are of the same dynamic range, but may break down in other cases: for example, diagonal entries in a matrix multiplication layer inside LSTM may have very different statistics comparing to non diagonal entries.
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
Shifted and Squeezed 8-bit Floating Point format for Low-Precision Training of Deep Neural Networks
Training with larger number of parameters while keeping fast iterations is an increasingly adopted strategy and trend for developing better performing Deep Neural Network (DNN) models. This necessitates increased memory footprint and computational requirements for training. Here we introduce a novel methodology for training deep neural networks using 8-bit floating point (FP8) numbers. Reduced bit precision allows for a larger effective memory and increased computational speed. We name this method Shifted and Squeezed FP8 (S2FP8). We show that, unlike previous 8-bit precision training methods, the proposed method works out of the box for representative models: ResNet50, Transformer and NCF. The method can maintain model accuracy without requiring fine-tuning loss scaling parameters or keeping certain layers in single precision. We introduce two learnable statistics of the DNN tensors - shifted and squeezed factors that are used to optimally adjust the range of the tensors in 8-bits, thus minimizing the loss in information due to quantization.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



The paper suggest the method to train neural networks using 8 bit floating point precision values. The method requires a hardware modification to evaluate per tensor statistics, which intuitively capture dynamic range that a given tensor values may fall into. Those per tensor statistics are used to map 8-bit representation into a rational number. Statistics themselves are calculated using 32 bit floating point numbers. While the method is interesting, I do not think it is practical due to the required hardware modifications. I am by no means not a hardware design expert, but I am not convinced that the gain of using 8 vs 16 bit floating point numbers outweights any extra complexity of hardware implementation. Mapping between representation and actual numerical values is much more complex than when using standard floating point representation (as any given numerical value is defined by a single its 8 bit representation + alpha + beta tensor statistics), integrated circuitry used to execute arithmetic operations would likely be much more complex than when using standard floating point operations. It is plausible that the added complexity (in terms of transistor count) would negate any potential price or energy savings over simply using 16 bit floating point representation. This should definitely be discussed in depth in this paper where the main contribution is an algorithm to be implemented in hardware. As different layers will have different (alpha,beta) tensor statistics, from the computational perspective they would look like different data types which would be cast into the same common type before doing any arithmetic operations on them. This would almost certainly require extra computational steps to be done potentially negating computational benefits over simply using 16 bit floating point numbers. The algorithm evaluates per-tensor and not per-tensor-element statistics. This may work well in cases where all entries in a given tensor are of the same dynamic range, but may break down in other cases: for example, diagonal entries in a matrix multiplication layer inside LSTM may have very different statistics comparing to non diagonal entries.
assistant
The paper suggest the method to train neural networks using 8 bit floating point precision values. The method requires a hardware modification to evaluate per tensor statistics, which intuitively capture dynamic range that a given tensor values may fall into. Those per tensor statistics are used to map 8-bit representation into a rational number. Statistics themselves are calculated using 32 bit floating point numbers. While the method is interesting, I do not think it is practical due to the required hardware modifications. I am by no means not a hardware design expert, but I am not convinced that the gain of using 8 vs 16 bit floating point numbers outweights any extra complexity of hardware implementation. Mapping between representation and actual numerical values is much more complex than when using standard floating point representation (as any given numerical value is defined by a single its 8 bit representation + alpha + beta tensor statistics), integrated circuitry used to execute arithmetic operations would likely be much more complex than when using standard floating point operations. It is plausible that the added complexity (in terms of transistor count) would negate any potential price or energy savings over simply using 16 bit floating point representation. This should definitely be discussed in depth in this paper where the main contribution is an algorithm to be implemented in hardware. As different layers will have different (alpha,beta) tensor statistics, from the computational perspective they would look like different data types which would be cast into the same common type before doing any arithmetic operations on them. This would almost certainly require extra computational steps to be done potentially negating computational benefits over simply using 16 bit floating point numbers. The algorithm evaluates per-tensor and not per-tensor-element statistics. This may work well in cases where all entries in a given tensor are of the same dynamic range, but may break down in other cases: for example, diagonal entries in a matrix multiplication layer inside LSTM may have very different statistics comparing to non diagonal entries.