Fully Quantized Transformer for Improved Translation
State-of-the-art neural machine translation methods employ massive amounts of parameters. Drastically reducing computational costs of such methods without affecting performance has been up to this point unsolved. In this work, we propose a quantization strategy tailored to the Transformer architecture. We evaluate our method on the WMT14 EN-FR and WMT14 EN-DE translation tasks and achieve state-of-the-art quantization results for the Transformer, obtaining no loss in BLEU scores compared to the non-quantized baseline. We further compress the Transformer by showing that, once the model is trained, a good portion of the nodes in the encoder can be removed without causing any loss in BLEU.
This paper proposes a method for reducing the required memory space by a quantization technique. This paper specifically focuses on reducing it for the Transformer architecture. Several similar methods that shared the same motivation have already been proposed in the literature. Moreover, the main idea of the proposed method is the uniform quantization method, which has described in the previous study. Therefore, unfortunately, the technical novelty seems to be very marginal and narrow. It is hard to say that this paper is innovative or has enough contribution and influence on the community. Please clearly elaborate if the authors think that the proposed method has certain technical novelty, and I missed to recognize it. 1, However, I am a bit surprised by the results of the proposed method; it successfully achieved the on-par result (or even improving) to the standard (non-quantized) models on the well-studied WMT14 EN-FR and WMT14 EN-DE translation tasks. According to Tables 1 and 2, it reduced the required memory space 1/4 from the original while maintaining any performance degradation, or even it slightly improves the performance. It is good to see such a simple approach offers a better result. I think that the experimental results may offer a new insight into the network quantization technique. 2, The paper evaluates the proposed method only on WMT14 En-De and En-Fr benchmark datasets. The effectiveness of the proposed method would be more convincing if it was evaluated on different datasets and tasks as well, e.g. wikitext-103 for language modeling. 3, Not much insight and discussion are provided on why the proposed method works better than other baselines. For example, the ablation study offers very interesting observations, but the authors only state the fact, and not providing any hypotheses for further analyzing the reason of working well for the proposed method. 4, One additional thing that I concern about this paper is the reproducibility of the experiments. I could not find a detailed explanation of how the authors obtained the reported results written in the tables. If the results were obtained from a single run for each method with the single random initialization, then it may be a bit unreliable. It is because we often observe more than 1point BLEU difference in the same model configuration except for a random initialization. To clear such suspicion, the authors should report the average BLEU of the several runs or release their code for reproducing their experiments in future validation.
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
Fully Quantized Transformer for Improved Translation
State-of-the-art neural machine translation methods employ massive amounts of parameters. Drastically reducing computational costs of such methods without affecting performance has been up to this point unsolved. In this work, we propose a quantization strategy tailored to the Transformer architecture. We evaluate our method on the WMT14 EN-FR and WMT14 EN-DE translation tasks and achieve state-of-the-art quantization results for the Transformer, obtaining no loss in BLEU scores compared to the non-quantized baseline. We further compress the Transformer by showing that, once the model is trained, a good portion of the nodes in the encoder can be removed without causing any loss in BLEU.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



This paper proposes a method for reducing the required memory space by a quantization technique. This paper specifically focuses on reducing it for the Transformer architecture. Several similar methods that shared the same motivation have already been proposed in the literature. Moreover, the main idea of the proposed method is the uniform quantization method, which has described in the previous study. Therefore, unfortunately, the technical novelty seems to be very marginal and narrow. It is hard to say that this paper is innovative or has enough contribution and influence on the community. Please clearly elaborate if the authors think that the proposed method has certain technical novelty, and I missed to recognize it. 1, However, I am a bit surprised by the results of the proposed method; it successfully achieved the on-par result (or even improving) to the standard (non-quantized) models on the well-studied WMT14 EN-FR and WMT14 EN-DE translation tasks. According to Tables 1 and 2, it reduced the required memory space 1/4 from the original while maintaining any performance degradation, or even it slightly improves the performance. It is good to see such a simple approach offers a better result. I think that the experimental results may offer a new insight into the network quantization technique. 2, The paper evaluates the proposed method only on WMT14 En-De and En-Fr benchmark datasets. The effectiveness of the proposed method would be more convincing if it was evaluated on different datasets and tasks as well, e.g. wikitext-103 for language modeling. 3, Not much insight and discussion are provided on why the proposed method works better than other baselines. For example, the ablation study offers very interesting observations, but the authors only state the fact, and not providing any hypotheses for further analyzing the reason of working well for the proposed method. 4, One additional thing that I concern about this paper is the reproducibility of the experiments. I could not find a detailed explanation of how the authors obtained the reported results written in the tables. If the results were obtained from a single run for each method with the single random initialization, then it may be a bit unreliable. It is because we often observe more than 1point BLEU difference in the same model configuration except for a random initialization. To clear such suspicion, the authors should report the average BLEU of the several runs or release their code for reproducing their experiments in future validation.
assistant
This paper proposes a method for reducing the required memory space by a quantization technique. This paper specifically focuses on reducing it for the Transformer architecture. Several similar methods that shared the same motivation have already been proposed in the literature. Moreover, the main idea of the proposed method is the uniform quantization method, which has described in the previous study. Therefore, unfortunately, the technical novelty seems to be very marginal and narrow. It is hard to say that this paper is innovative or has enough contribution and influence on the community. Please clearly elaborate if the authors think that the proposed method has certain technical novelty, and I missed to recognize it. 1, However, I am a bit surprised by the results of the proposed method; it successfully achieved the on-par result (or even improving) to the standard (non-quantized) models on the well-studied WMT14 EN-FR and WMT14 EN-DE translation tasks. According to Tables 1 and 2, it reduced the required memory space 1/4 from the original while maintaining any performance degradation, or even it slightly improves the performance. It is good to see such a simple approach offers a better result. I think that the experimental results may offer a new insight into the network quantization technique. 2, The paper evaluates the proposed method only on WMT14 En-De and En-Fr benchmark datasets. The effectiveness of the proposed method would be more convincing if it was evaluated on different datasets and tasks as well, e.g. wikitext-103 for language modeling. 3, Not much insight and discussion are provided on why the proposed method works better than other baselines. For example, the ablation study offers very interesting observations, but the authors only state the fact, and not providing any hypotheses for further analyzing the reason of working well for the proposed method. 4, One additional thing that I concern about this paper is the reproducibility of the experiments. I could not find a detailed explanation of how the authors obtained the reported results written in the tables. If the results were obtained from a single run for each method with the single random initialization, then it may be a bit unreliable. It is because we often observe more than 1point BLEU difference in the same model configuration except for a random initialization. To clear such suspicion, the authors should report the average BLEU of the several runs or release their code for reproducing their experiments in future validation.