Published as a conference paper at ICLR 2018 UNSUPERVISED CIPHER CRACKING USING DISCRETE GANS
This work details CipherGAN, an architecture inspired by CycleGAN used for inferring the underlying cipher mapping given banks of unpaired ciphertext and plaintext. We demonstrate that CipherGAN is capable of cracking language data enciphered using shift and Vigenère ciphers to a high degree of fidelity and for vocabularies much larger than previously achieved. We present how CycleGAN can be made compatible with discrete data and train in a stable way. We then prove that the technique used in CipherGAN avoids the common problem of uninformative discrimination associated with GANs applied to discrete data.
SUMMARY
The paper considers the problem of using cycle GANs to decipher text encrypted with historical ciphers. Also it presents some theory to address the problem that discriminating between the discrete data and continuous prediction is too simple. The model proposed is a variant of the cycle GAN in which in addition embeddings helping the Generator are learned for all the values of the discrete variables. 
The log loss of the GAN is replaced by a quadratic loss and a regularization of the Jacobian of the discriminator. Experiments show that the method is very effective.  
REVIEW
The paper considers an interesting and fairly original problem and the overall discussion of ciphers is quite nice. Unfortunately, my understanding is that the theory proposed in section 2 does not correspond to the scheme used in the experiments (contrarily to what the conclusion suggest and contrarily to what the discussion of the end of section 3, which says that using embedding is assumed to have an equivalent effect to using the methodology considered in the theoretical part). Another important concern is with the proof: there seems to be an unmotivated additional assumption that appears in the middle of the proof of Proposition 1 + some steps need to be clarified (see comment 16 below).
The experiments do not have any simple baseline, which is somewhat unfortunate.
DETAILED COMMENTS:
1- The paper makes a few bold and debatable statements:
line 9 of section 1
"Such hand-crafted features have fallen out of favor (Goodfellow et al., 2016) as a
result of their demonstrated inferiority to features learned directly from data in end-to-end learning
frameworks such as neural networks"
This is certainly an overstatement and although it might be true for specific types of inputs it is not universally true, most deep architectures rely on a human-in-the-loop and there are number of areas where human crafted feature are arguably still relevant, if only to specify what is the input of a deep network: there are many domains where the notion of raw data does not make sense, and, when it does, it is usually associated with a sensing device that has been designed by a human and which implicitly imposes what the data is based on human expertise. 
2- In the last paragraph of the introduction, the paper says that previous work has only worked on vocabularies of 26 characters while the current paper tackles word level ciphers with 200 words. But, isn't this just a matter of scalability and only possible with very large amounts of text? Is it really because of an intrinsic limitation or lack of scalability of previous approaches or just because the authors of the corresponding papers did not care to present larger scale experiments? 
3- The discussion at the top of page 5 is difficult to follow. What do you mean when you say "this motivates the benefits of having strong curvature globally, as opposed to linearly between etc"
Which curvature are we talking about? and what how does the "as opposed to linearly" mean? Should we understand "as opposed to having curvature linearly interpolated between etc" or "as opposed to having a linear function"? Please clarify.
4- In the same paragraph: what does "a region that has not seen the Jacobian norm applied to it" mean? How is a norm applied to a region? I guess that what you mean is that the generator G might creates samples in a part of the space where the function F has not yet been learned and is essentially close to 0. Is this what you mean?
5- I do not understand why the paper introduces WGAN since in the end it does not use them but uses a quadratic loss, introduced in the first display of section 4.3.
6- The paper makes a theoretical contribution which supports replacing the sample y by a sample drawn from a region around y. But it seems that this is not used in the experiment and that the authors consider that the introduction of the embedding is a substitution for this. Indeed, in the last paragraph of section 3.1, the paper says "we make the assumption that the training of the embedding vectors approximates random sampling similar to what is described in Proposition 1". This does not make any sense to me because the embedding vectors map each y deterministically to a single point, and so the distribution on the corresponding vectors is still a fixed discrete distribution. This gives me this impression that the proposed theory does not match what is used in the experiments.
(The last sentence of section 3.1, which is commenting on this and could perhaps clarify the situation is ill formed with two verbs.)
7- In the definitions: "A discriminator is said to perform uninformative discrimination" etc. -> It seems that the choice of the word uninformative would be misleading: an uninformative discrimination would be a discrimination that completely fails, while what the condition is saying it that it cannot perform perfect discrimination. I would thus suggest to call this "imperfect discrimination". 
8- It seems that the same embedding is used in X space and in Y space (from equations 6 and 7). Is there any reason for that? I would seem more natural to me to introduce two different embeddings since the objects are a priori different...
Actually I don't understand how the embeddings can be the same in the Vignere code case since time taken into account one one side.
9- On the 5th line after equation (7), the paper says "the embeddings... are trained to minimize L_GAN and L_cyc, meaning... and are easy to discriminate" -> This last part of the sentence seems wrong to me. The discriminator is trying to maximize L_GAN and so minimizing w.r.t. to the embedding is precisely trying to prevent to the discriminator to tell apart too easily the true elements from the estimated ones.
In fact the regularization of the Jacobian that will be preventing the discriminator to vary too quickly in space is more likely to explain the fact that the discrimination is not too easy to do between the true and mapped embeddings. This might be connected to the discussion at the top of page 5. Since there are no experiments with alpha different than the default value = 10, this is difficult to assess.
10-The Vigenere cipher is explained again at the end of section 4.2 when it has already been presented in section 1.1
11- Concerning results in Table 2: I do not see why it would not be possible to compare the performance of the method with classical frequency analysis, at least for the character case.
12- At the beginning of section 4.3, the text says that the log loss was replaced with the quadratic loss, but without giving any reason. Could you explain why.
13- The only comparison of results with and without embeddings is presented in the curves of figure 3, for Brown-W with a vocabulary of 200 words. In that case it helps. Could the authors report systematically results about all cases? (I guess this might however be the only hard case...)
14- It would be useful to have a brief reminder of the architecture of the neural network (right now the reader is just refered to Zhu et al., 2017): how many layers, how many convolution layers etc.
The same comment applies for the way the position of the letter/word in the text appear is in encoded in a feature that is provided as input to the neural network: it would be nice if the paper could provide a few details here and be more self contained. (The fact that the engineering of the time feature can "dramatically" improve the performance of the network should be an argument to convince the authors that hand-crafted feature have not fallen out of favor completely yet...)
15- I disagree with the statement made in the conclusion that the proposed work "empirically confirms [...] that the use of continuous relaxation of discrete variable facilitates [...] and prevents [...]" because for me the proposed implementation does not use at all the theoretical idea of continuous relaxation proposed in the paper, unless there is a major point that I am missing.
16- I have two issues with the proof in the appendix
a) after the first display of the last page the paper makes an additional assumption which is not announced in the statement of the theorem, which is that two specific inequality hold...
Unless I am mistaken this assumption is never proven (later or earlier). Given that this inequality is just "the right inequality to get the proof go through" and given that there are no explanation for why this assumption is reasonable, to me this invalidates the proof. The step of going from G(S_y) to S_(G(y)) seems delicate...
b) If we accept these inequalities, the determinant of the Jacobian (the notation is not defined) of F at (x_bar) disappears from the equations, as if it could be assumed to be greater than one. If this is indeed the case, please provide a justification of this step.
17- A way to address the issue of trivial discrimination in GANs with discrete data has been proposed in
 
Luc, P., Couprie, C., Chintala, S., & Verbeek, J. (2016). Semantic segmentation using adversarial networks. arXiv preprint arXiv:1611.08408.
The authors should probably reference this paper.
18- Clarification of the Jacobian regularization: in equation (3), the Jacobian computed seems to be w.r.t D composed with F while in equation (8) it is only the Jacobian of D. Which equation is the correct one?
TYPOS:
Proposition 1: the if-then statement is broken into two sentences separated by a full point and a carriage return.
sec. 4.3 line 10 we use a cycle loss *with a regularization coefficient* lambda=1 (a piece of the sentence is missing)
sec. 4.3 lines 12-13 the learning rates given are the same at startup and after "warming up"...
In the appendix: 
3rd line of proof of prop 1: I don' understand "countably infinite finite sequences of vectors lying in the vertices of the simplex" -> what is countable infinite here? The vertices?