Deepcode: Feedback Codes via Deep Learning
The design of codes for communicating reliably over a statistically well defined channel is an important endeavor involving deep mathematical research and wideranging practical applications. In this work, we present the first family of codes obtained via deep learning, which significantly beats state-of-the-art codes designed over several decades of research. The communication channel under consideration is the Gaussian noise channel with feedback, whose study was initiated by Shannon; feedback is known theoretically to improve reliability of communication, but no practical codes that do so have ever been successfully constructed. We break this logjam by integrating information theoretic insights harmoniously with recurrent-neural-network based encoders and decoders to create novel codes that outperform known codes by 3 orders of magnitude in reliability. We also demonstrate several desirable properties in the codes: (a) generalization to larger block lengths; (b) composability with known codes; (c) adaptation to practical constraints. This result also presents broader ramifications to coding theory: even when the channel has a clear mathematical model, deep learning methodologies, when combined with channel-specific information-theoretic insights, can potentially beat state-of-the-art codes, constructed over decades of mathematical research.
I acknowledge that I read the author's response, which address many of my concerns. As a result, I am raising my score to a 5 -- I lean toward rejection but not strongly. I remain concerned about the absence of a related work section -- and the related work cited in the response still seems inadequate. In our discussions, R2 pointed out a lot of additional work the authors appear to have missed.  -----  Summary:  This paper describes a deep learning approach to encoding and decoding in a noisy channel with feedback, something that has been little explored in the literature to date. The formal noisy channel setting is similar to a standard autoencoder framework, with a few key differences. For one, we usually encode and transmit one bit of a message at time due to channel limits, and second we get feedback, usually in the form of a noisy version of each encoded bit. Due to the sequential nature of the problem, plus the availability of feedback, the authors apply an RNN architecture. The input to the decoder at each step is the next bit to encode plus an estimate of the noise from previous steps (derived from the difference between the encoded message and the received feedback). Experiments suggest that this approach significantly outperforms existing approaches. The authors explore several variants of the RNN architecture specific to the setting: normalization to restrict the power of the encoded message, per-bit weights to adapt the power of each encoded message (similar to an attention mechanism), and an encoder-decoder framework for the feedback.  On the one hand, I think this paper is pretty solid and could be accepted on the merits of its work.  On the other hand, the paper lacks entirely a related work section and fails to discuss ANY previous work on deep learning applied to noisy channels. A quick Google search indicates there is plenty of such work (as does the mysterious presence of uncited papers in the references, see below).  It is hard to imagine accepting the paper in its current form. I would like to hear the authors' explanation and discuss with the other reviewers before making a final decision. For now I will put a temporary "clear reject" score but am willing to revise it.  Quality:  Although a straightforward application for the most part, the work nonetheless appears high quality to me. The choice to apply an RNN seems sound and the various proposed architectures seem well-matched to the problem. The authors demonstrate a thorough understanding of the problem setting (though their description could be improved, see below). The results appear to speak for themselves: this approach soundly beats the baselines.  I especially appreciate Sections 4 and 5 -- the experiments in this paper are thorough, and the authors make a good faith effort to provide insights about their proposed model's behavior, rather than just reporting quantitative performance.  A few questions about the proposed approach and experiments:  - The authors need to justify the choice to use an RNN without first experimenting with a non-recurrent architecture. Do we really need to capture longterm dependencies? The coupling results (lines 313-318) only cover a very short history: we could easily design an MLP or convolutional architecture that accepts multiple steps of input to capture such history. Did the authors try this?  - The discussion of complexity (lines 234-243) is somewhat vague. I expected to see an actual complexity analysis (in terms of block length and maybe coded message length?) and then maybe some empirical timing experiments -- along with a comparison to existing algorithms (in both formal analysis and in the experiments). Instead we got a vague handwave suggesting that they're comparable. That's not very convincing.  Clarity:  This paper is for the most part well-organized and written (modulo the missing related work), but it may be difficult for readers who are unfamiliar with coding theory. To make it more accessible to a wider reader audience (thus increasing its potential impact), I suggest the following:  - clearly articulate how this problem differs from, say, standard autoencoding problems or machine translation, which both have similar looking structure.  - show the full end-to-end architecture with encoder AND decoder and explain how it works. It's still not clear to me whether the decoder is run once at the end, after the full coded message is transmitted (as implied by the bidirectional RNN architecture) or whether each decoding step happens after the corresponding encoded step is transmitted.  Some nitpicks:  - The justification for the first phase is hard to follow. Why do we want to send the un-encoded bits first? What would happen if we omitted that phase (maybe some results for an RNN with phase 1)?  - Why does the RNN in Figure 2 output two coded bits per step? Is that simply due to the rate of 1/3 per the equation in lines 164-165 -- so that with a different rate, we'd output more codes?  - Why is the input to the RNN at each step the difference between the feedback and codes, rather than just the raw feedback? Did the authors try this?  - What is meant by "unrolling" the RNN cells when working with longer block lengths (line 284)? Does this mean using truncated backprop through time or what?  - The description of the architecture used (lines 239-241) is a bit confusing: the authors say they used an RNN in the encoder but a GRU in the decoder. Is the encoder in fact a vanilla RNN or is this a typo?  Originality:  It is impossible to judge the paper's originality in the absence of a thorough discussion of related work. Why is there no related work section, or at least a discussion of relevant previous work in the introduction? This is especially baffling given that the references section lists multiple relevant papers [10-17] that are not cited anywhere in the paper. This means they're listed in the compiled bibtex database file, which in turn means that at some point, a version of the paper citing them must have been compiled. Was the related work section accidentally commented out? On the other hand, the paper is exactly 8 pages long, meaning that there's no room for related work anyway.  In any case, the paper's language, the lack of discussion of similar work, and the absence of other neural net baselines, implies this work may be the first to apply RNNs to the noisy channel with feedback problem. I am not sufficiently familiar with work in this area to confirm or deny that claim, but I am dubious of any claim to be the first to apply deep learning to a classic problem -- and the works listed in the references in this paper [10-17] (plus those turned up in a quick search) cast some doubt.  The authors need to tighten their claim: what exactly is their original contribution? And they need to position their work with respect to related research. If there are alternative approaches to this problem -- either previously published or even not published but obvious -- then they need to make a best effort to compare them rhetorically and empirically.  Significance:  The significance of an applied paper of this type is tied almost entirely to its empirical results and what new insights we can derive from them. Taking the paper at face value, it appears quite significant -- the proposed RNN architecture is substantially better than standard approaches, and I -- for one -- did not know that, even if it seems unsurprising.  However, given the omission of a related work discussion, it is impossible to fully decide the significance of this paper. If someone else previously applied RNNs to this problem and produced similar results, then this paper's contribution is diminished.