Lossless Data Compression with Transformer
Transformers have replaced long-short term memory and other recurrent neural networks variants in sequence modeling. It achieves state-of-the-art performance on a wide range of tasks related to natural language processing, including language modeling, machine translation, and sentence representation. Lossless compression is another problem that can benefit from better sequence models. It is closely related to the problem of online learning of language models. But, despite this ressemblance, it is an area where purely neural network based methods have not yet reached the compression ratio of state-of-the-art algorithms. In this paper, we propose a Transformer based lossless compression method that match the best compression ratio for text. Our approach is purely based on neural networks and does not rely on hand-crafted features as other lossless compression algorithms. We also provide a thorough study of the impact of the different components of the Transformer and its training on the compression ratio.
This paper provides a method for lossless compression of text. It*s heavily inspired by the language modelling methods that have been developed for the purposes of predicting the next character/word in a sentence, and it uses this idea as its backbone. The only difference is that the results are presented in the compression setting. I think we should reject this paper due to the following reasons: - I don*t see enough of a difference between this and previous work - the results are nowhere near SoTA for compression, despite the method being sold to this community - there are other papers that do lossless neural compression that could have been used to make a comparison rather than making no comparison at all. For example, *Practical Full Resolution Learned Lossless Image Compression* (CVPR 2019) provides a framework for image rather than text, but that could be adapted to this field without any major changes (predict convolutionally characters, rather than RGB values). - there*s no comparison even with BERT (how well it do to predict the next character vs. this)... - no runtime numbers - no reproducibility discussion (i.e., how can I guarantee that my decoder can get exactly the same numbers as my encoder so that I can decompress on a different machine) - no discussion about whether files were created/decompressed (this is ABSOLUTELY CRUCIAL for compression papers to discuss) Overall, I am not excited about this paper, and unless the authors put a lot more into it, there*s just not enough novelty to justify a publication at ICLR.
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
Lossless Data Compression with Transformer
Transformers have replaced long-short term memory and other recurrent neural networks variants in sequence modeling. It achieves state-of-the-art performance on a wide range of tasks related to natural language processing, including language modeling, machine translation, and sentence representation. Lossless compression is another problem that can benefit from better sequence models. It is closely related to the problem of online learning of language models. But, despite this ressemblance, it is an area where purely neural network based methods have not yet reached the compression ratio of state-of-the-art algorithms. In this paper, we propose a Transformer based lossless compression method that match the best compression ratio for text. Our approach is purely based on neural networks and does not rely on hand-crafted features as other lossless compression algorithms. We also provide a thorough study of the impact of the different components of the Transformer and its training on the compression ratio.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



This paper provides a method for lossless compression of text. It*s heavily inspired by the language modelling methods that have been developed for the purposes of predicting the next character/word in a sentence, and it uses this idea as its backbone. The only difference is that the results are presented in the compression setting. I think we should reject this paper due to the following reasons: - I don*t see enough of a difference between this and previous work - the results are nowhere near SoTA for compression, despite the method being sold to this community - there are other papers that do lossless neural compression that could have been used to make a comparison rather than making no comparison at all. For example, *Practical Full Resolution Learned Lossless Image Compression* (CVPR 2019) provides a framework for image rather than text, but that could be adapted to this field without any major changes (predict convolutionally characters, rather than RGB values). - there*s no comparison even with BERT (how well it do to predict the next character vs. this)... - no runtime numbers - no reproducibility discussion (i.e., how can I guarantee that my decoder can get exactly the same numbers as my encoder so that I can decompress on a different machine) - no discussion about whether files were created/decompressed (this is ABSOLUTELY CRUCIAL for compression papers to discuss) Overall, I am not excited about this paper, and unless the authors put a lot more into it, there*s just not enough novelty to justify a publication at ICLR.
assistant
This paper provides a method for lossless compression of text. It*s heavily inspired by the language modelling methods that have been developed for the purposes of predicting the next character/word in a sentence, and it uses this idea as its backbone. The only difference is that the results are presented in the compression setting. I think we should reject this paper due to the following reasons: - I don*t see enough of a difference between this and previous work - the results are nowhere near SoTA for compression, despite the method being sold to this community - there are other papers that do lossless neural compression that could have been used to make a comparison rather than making no comparison at all. For example, *Practical Full Resolution Learned Lossless Image Compression* (CVPR 2019) provides a framework for image rather than text, but that could be adapted to this field without any major changes (predict convolutionally characters, rather than RGB values). - there*s no comparison even with BERT (how well it do to predict the next character vs. this)... - no runtime numbers - no reproducibility discussion (i.e., how can I guarantee that my decoder can get exactly the same numbers as my encoder so that I can decompress on a different machine) - no discussion about whether files were created/decompressed (this is ABSOLUTELY CRUCIAL for compression papers to discuss) Overall, I am not excited about this paper, and unless the authors put a lot more into it, there*s just not enough novelty to justify a publication at ICLR.