Answers.Body: <p>The answer to your question depends on the nature of the noise that you have xor-ed the image with. If it is the case that the noise is random (or <a href="https://en.wikipedia.org/wiki/Pseudorandomness" rel="nofollow noreferrer">pseudorandom</a> in the formal sense), then it is provably the case that the original pattern will not be learnable in the statistical learning theory sense; this scenario is equivalent to the application of a <a href="https://en.wikipedia.org/wiki/One-time_pad" rel="nofollow noreferrer">one-time pad</a>.</p>
  
  <p>To quote the relevant Wikipedia article:</p>
  
  <blockquote>
    <p>One-time pads are "information-theoretically secure" in that the encrypted message (i.e., the ciphertext) provides no information about the original message to a cryptanalyst (except the maximum possible length[16] of the message). This is a very strong notion of security first developed during WWII by Claude Shannon and proved, mathematically, to be true for the one-time pad by Shannon about the same time.</p>
  </blockquote>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-11-11T05:40:05.210Z
Answers.Text: Hi eric! I have generated a (pseudo)random key of the same length as the image with values in the range of 0-255 and then applied it to every single image in the dataset. From my perspective, because the network would see hundreds of thousands of images encoded with the same key, it could draw conclusions about which encoded pixels correlate with other encoded pixels in a sequence of images. But, as Neil mentioned above, CNN is not the right way to go on that, because optically, the noise doesn't make sense.
Answers.UserDisplayName: Filip Dziuba
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-11-11T01:08:36.100Z
Answers.LastActivityDate: 2017-11-11T01:08:36.100Z
Answers.UserDisplayName: eric.mitchell
Answers.Body: <p>My gut feel based on the paper I will mention below is that <strong>YES</strong>, if you apply the same XOR operation on the train and test data, you will be able to train a very "accurate" classifier. </p>
  
  <p>To elaborate on my "gut" feel, please allow me to introduce to you what I personally think is one of the most important paper that came out this year(in fact this paper won the best paper award at ICLR 2017):</p>
  
  <p><a href="https://arxiv.org/abs/1611.03530" rel="nofollow noreferrer"> Understanding deep learning requires rethinking generalization</a>. </p>
  
  <p>In this paper, the authors showed that deep learning models will generalize to "any" datasets. To give an example of the sort of experiment they conducted on this paper: </p>
  
  <ul>
  <li>They randomly shuffled the training and test set's labels around in such a manner that for example some images of cats were labeled as dogs whiles some dogs were named cats whilst some cats and dogs images remained correctly labeled. Now it is well understood that deep learning models(including CNNs) are quite resistant to a few noisy labels but in the experiments conducted in the paper mentioned above this was a significant amount of noisy which begs the question why neural networks still performed well on what ended up being a garbage dataset. </li>
  </ul>
  
  <p>The moral of the story is that contrary to what most researchers believed in the past namely that deep learning models magically discover lower level features, middle-level features, and higher-level features hidden within the dataset more like the V1 system of the mammalian brain by learning to compress data, they seem to just memorize anything you give them, including random data. </p>
  
  <p>In short the paper mentioned above showed that deep learning models generalize well to completely random noise(in your case, think images generated from random pixels). Deep learning models will generalize well to anything, anything. And if they can generalize to random data which have no structure, then images that underwent a fixed, predefined transformation like XOR have nothing to a deep learning model. </p>
  
  <p><em>I must say, this are very worrying findings - to me at least.</em> </p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-11-11T18:59:40.683Z
Answers.LastActivityDate: 2017-11-11T18:59:40.683Z
Answers.UserDisplayName: Tshilidzi Mudau
Answers.Body: <p>First you should give it a try, because anyone's guess could be off, as there isn't really a complete high level analytic model of how neural networks behave on real data. Most results with neural networks are informed by theory, but involve a whole lot of experimental testing.</p>
  
  <p>I suspect your network might at least learn <em>something</em> from the new images, but would struggle to get anything like the same accuracy as without the noise, because the CNN filters rely on being able to detect similar features at different positions. In your scrambled image, there will not be any meaningful and <em>consistent</em> edges/corners etc that a single learned feature detector could learn to match (and therefore present to the next layer).</p>
  
  <p>A fully-connected network would not have this limitation, and would learn just as well on a set of <em>binary</em> features that have been xor'd identically in each position for each example, as it did on the original copy (i.e. only if the picture was 1 bit depth). It would learn less well if each feature was a scaled 8-bit pixel value that was xor'd with the same 8 bit random number in each pixel position, because that would introduce many more non-linear mappings between input and output. Of course a fully-connected network will generally not learn image tasks as well as CNNs in the first place . . . but if it could learn anything useful at all for your image problem, then it will probably out-perform CNNs after the scrambling effect.</p>
  
  <p>As a CNN usually has a few fully-connected layers, then it may be possible to get <em>something</em> from your scrambled images.</p>
  
  <blockquote>
    <p>Thinking logically, the features still exist in the same relation to each other, just in a different form (encoded)</p>
  </blockquote>
  
  <p>In terms of being recognisable in the way that a CNN filter extracts them, then the features do not exist. That is a problem.</p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-11-11T05:29:27.857Z
Answers.Text: Thank you for your answer, Neil. I tried some various CNN setups and, as expected, none were able to converge on the problem. That was very insightful for me, as my journey with machine learning just started. To respond to your suggestion concerning a fully-connected network, the beauty of CNN's is that different positions of the features don't hinder the network from performing. So even if a fully-connected structure would solve the problem of accurately representing information from the image, it would make the classification task undoable due to the variance of the training set.
Answers.UserDisplayName: Filip Dziuba
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-11-10T22:31:11.353Z
Answers.LastActivityDate: 2017-11-12T09:04:28.110Z
Answers.LastEditDate: 2017-11-12T09:04:28.110Z
Answers.UserDisplayName: Neil Slater
Body: <p>I have a set of images that I already trained a CNN to classify successfully. I wonder if it would be possible to encode the images (using XOR in combination with a key of the same length as the image) and train a new net on them. </p>
  
  <p>Thinking logically, the features still exist in the same relation to each other, just in a different form (encoded). Considering that neural networks are incredible at pattern recognition, I assume that it would still be doable.</p>
  
  <p>For people, who cannot imagine how a xor-encoded image would look like:
  <a href="https://i.sstatic.net/GsIxN.png" rel="noreferrer"><img src="https://i.sstatic.net/GsIxN.png" alt="example for encoded image using random key"></a></p>
  
  <p>For a human, it may look like rubbish, but the information is definitely there.</p>
  
  <p>Would love to read your opinion.</p>
  
Tags: <neural-networks><convolutional-neural-networks><image-recognition>
Title: Would convolutional NN recognize patterns in encoded images?
UserDisplayName: Filip Dziuba
ThreadText: # Would convolutional NN recognize patterns in encoded images?
  
  (Asked by: username_0 on 2017-11-09)
  
  I have a set of images that I already trained a CNN to classify successfully. I wonder if it would be possible to encode the images (using XOR in combination with a key of the same length as the image) and train a new net on them.
  
  Thinking logically, the features still exist in the same relation to each other, just in a different form (encoded). Considering that neural networks are incredible at pattern recognition, I assume that it would still be doable.
  
  For people, who cannot imagine how a xor-encoded image would look like:
  [![example for encoded image using random key](https://i.sstatic.net/GsIxN.png)](https://i.sstatic.net/GsIxN.png)
  
  For a human, it may look like rubbish, but the information is definitely there.
  
  Would love to read your opinion.
  
  **Tags:** neural-networks, convolutional-neural-networks, image-recognition
  
  ---
  
  ## 3 answers
  
  ### Answer by username_3 on 2017-11-10. Score: 5 | Accepted answer
  
  First you should give it a try, because anyone's guess could be off, as there isn't really a complete high level analytic model of how neural networks behave on real data. Most results with neural networks are informed by theory, but involve a whole lot of experimental testing.
  
  I suspect your network might at least learn *something* from the new images, but would struggle to get anything like the same accuracy as without the noise, because the CNN filters rely on being able to detect similar features at different positions. In your scrambled image, there will not be any meaningful and *consistent* edges/corners etc that a single learned feature detector could learn to match (and therefore present to the next layer).
  
  A fully-connected network would not have this limitation, and would learn just as well on a set of *binary* features that have been xor'd identically in each position for each example, as it did on the original copy (i.e. only if the picture was 1 bit depth). It would learn less well if each feature was a scaled 8-bit pixel value that was xor'd with the same 8 bit random number in each pixel position, because that would introduce many more non-linear mappings between input and output. Of course a fully-connected network will generally not learn image tasks as well as CNNs in the first place . . . but if it could learn anything useful at all for your image problem, then it will probably out-perform CNNs after the scrambling effect.
  
  As a CNN usually has a few fully-connected layers, then it may be possible to get *something* from your scrambled images.
  
  > Thinking logically, the features still exist in the same relation to each other, just in a different form (encoded)
  
  In terms of being recognisable in the way that a CNN filter extracts them, then the features do not exist. That is a problem.
  
  > **Comments:**
  > **username_0**: Thank you for your answer, Neil. I tried some various CNN setups and, as expected, none were able to converge on the problem. That was very insightful for me, as my journey with machine learning just started. To respond to your suggestion concerning a fully-connected network, the beauty of CNN's is that different positions of the features don't hinder the network from performing. So even if a fully-connected structure would solve the problem of accurately representing information from the image, it would make the classification task undoable due to the variance of the training set.  
  
  ---
  
  ### Answer by username_1 on 2017-11-11. Score: 1
  
  The answer to your question depends on the nature of the noise that you have xor-ed the image with. If it is the case that the noise is random (or [pseudorandom](https://en.wikipedia.org/wiki/Pseudorandomness) in the formal sense), then it is provably the case that the original pattern will not be learnable in the statistical learning theory sense; this scenario is equivalent to the application of a [one-time pad](https://en.wikipedia.org/wiki/One-time_pad).
  
  To quote the relevant Wikipedia article:
  
  > One-time pads are "information-theoretically secure" in that the encrypted message (i.e., the ciphertext) provides no information about the original message to a cryptanalyst (except the maximum possible length[16] of the message). This is a very strong notion of security first developed during WWII by Claude Shannon and proved, mathematically, to be true for the one-time pad by Shannon about the same time.
  
  > **Comments:**
  > **username_0**: Hi eric! I have generated a (pseudo)random key of the same length as the image with values in the range of 0-255 and then applied it to every single image in the dataset. From my perspective, because the network would see hundreds of thousands of images encoded with the same key, it could draw conclusions about which encoded pixels correlate with other encoded pixels in a sequence of images. But, as Neil mentioned above, CNN is not the right way to go on that, because optically, the noise doesn't make sense.  
  
  ---
  
  ### Answer by username_2 on 2017-11-11. Score: 0
  
  My gut feel based on the paper I will mention below is that **YES**, if you apply the same XOR operation on the train and test data, you will be able to train a very "accurate" classifier.
  
  To elaborate on my "gut" feel, please allow me to introduce to you what I personally think is one of the most important paper that came out this year(in fact this paper won the best paper award at ICLR 2017):
  
  [Understanding deep learning requires rethinking generalization](https://arxiv.org/abs/1611.03530).
  
  In this paper, the authors showed that deep learning models will generalize to "any" datasets. To give an example of the sort of experiment they conducted on this paper:
  
  * They randomly shuffled the training and test set's labels around in such a manner that for example some images of cats were labeled as dogs whiles some dogs were named cats whilst some cats and dogs images remained correctly labeled. Now it is well understood that deep learning models(including CNNs) are quite resistant to a few noisy labels but in the experiments conducted in the paper mentioned above this was a significant amount of noisy which begs the question why neural networks still performed well on what ended up being a garbage dataset.
  
  The moral of the story is that contrary to what most researchers believed in the past namely that deep learning models magically discover lower level features, middle-level features, and higher-level features hidden within the dataset more like the V1 system of the mammalian brain by learning to compress data, they seem to just memorize anything you give them, including random data.
  
  In short the paper mentioned above showed that deep learning models generalize well to completely random noise(in your case, think images generated from random pixels). Deep learning models will generalize well to anything, anything. And if they can generalize to random data which have no structure, then images that underwent a fixed, predefined transformation like XOR have nothing to a deep learning model.
  
  *I must say, this are very worrying findings - to me at least.*
  
  
  ---
  
  