Answers.Body: <p>Auto-encoders, a family of ANNs, are trained with exactly compression in mind. So definitely some ANNs are compressors.</p>
  
  <p>Also, in general, ANNs learn the best concepts to minimize fitness error. I would say that that means, both in classification and regression, to 1) differentiate between various inputs and 2) output the proper value for each input. Point 1), in particular, means having as many distinct net activation configurations as necessary to (at least) tell apart inputs needing (significantly, not in a statistical sense, just a qualitative sense) different outputs.
  I am inclined to think that if the input is complex enough, you would call what happens "compression".</p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2016-09-10T00:48:38.647Z
Answers.LastActivityDate: 2016-09-10T00:48:38.647Z
Answers.UserDisplayName: marcotama
Answers.Body: <p>ANNs don't compress, they generalise. </p>
  
  <p>Often this leads to compression, i.e. the internal generalised representation is smaller than the original input, but not necessarily. Imagine a ANN that is trained to use the screen input of a computer game to play it. If the game is very rich and conceptually deep the internal representation of a single screen input might be a lot bigger than the input itself, because ANNs put the single data points into the context of the overall data. Which leads us to the second point:</p>
  
  <p>ANNs (and the neocortex) model data hierarchically. </p>
  
  <p>This is what makes them so powerful. So it is not just about having a large number of parameters, they also have to be arranged in such a way that they capture the structure of the data (or the world), which very often seems to be hierarchical. Just look a two different pictures of a duck. On the pixel level they might be as different as random images, all the similarities emerge in higher levels of the hierarchy, when enough pixels combined give you the patterns of feathers, beak and webs. Bloom filters obviously lack this property. They would only give you a pixel by pixel account of whether you have seen (almost) exactly this picture before. </p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2016-09-09T23:56:59.743Z
Answers.Text: It looks like there is prior work in hierarchical data compression (see [here](http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.22.3180&rep=rep1&type=pdf)[pdf], [here](http://www.columbia.edu/~jw2966/papers/MDHW07-PAMI.pdf)[pdf] and [here](https://en.wikipedia.org/wiki/Set_partitioning_in_hierarchical_trees)).
Answers.UserDisplayName: Doxosophoi
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2016-09-08T11:36:01.787Z
Answers.LastActivityDate: 2016-09-08T11:36:01.787Z
Answers.UserDisplayName: BlindKungFuMaster
Body: <p>Conceptually speaking, aren't artificial neural networks just highly distributed, lossy compression schemes?</p>
  
  <p>They're certainly efficient at <a href="https://cs.stanford.edu/people/eroberts/courses/soco/projects/neural-networks/Applications/imagecompression.html" rel="nofollow">compressing images</a>.</p>
  
  <p>And aren't brains (at least, the neocortex) just compartmentalized, highly distributed, lossy databases?</p>
  
  <p>If so, what salient features in RNNs and CNNs are necessary in any given lossy compression scheme in order to extract the semantic relations that they do? Is it just a matter of having a large number of dimensions/variables? </p>
  
  <p>Could some kind of lossy <a href="https://en.wikipedia.org/wiki/Bloom_filter" rel="nofollow">Bloom filter</a> be re-purposed for the kinds of problems ANNs are applied to?</p>
  
Tags: <neural-networks>
Title: Are ANNs just highly distributed lossy compression schemes?
UserDisplayName: Doxosophoi
ThreadText: # Are ANNs just highly distributed lossy compression schemes?
  
  (Asked by: username_0 on 2016-09-08)
  
  Conceptually speaking, aren't artificial neural networks just highly distributed, lossy compression schemes?
  
  They're certainly efficient at [compressing images](https://cs.stanford.edu/people/eroberts/courses/soco/projects/neural-networks/Applications/imagecompression.html).
  
  And aren't brains (at least, the neocortex) just compartmentalized, highly distributed, lossy databases?
  
  If so, what salient features in RNNs and CNNs are necessary in any given lossy compression scheme in order to extract the semantic relations that they do? Is it just a matter of having a large number of dimensions/variables?
  
  Could some kind of lossy [Bloom filter](https://en.wikipedia.org/wiki/Bloom_filter) be re-purposed for the kinds of problems ANNs are applied to?
  
  **Tags:** neural-networks
  
  ---
  
  ## 2 answers
  
  ### Answer by username_1 on 2016-09-10. Score: 1 | Accepted answer
  
  Auto-encoders, a family of ANNs, are trained with exactly compression in mind. So definitely some ANNs are compressors.
  
  Also, in general, ANNs learn the best concepts to minimize fitness error. I would say that that means, both in classification and regression, to 1) differentiate between various inputs and 2) output the proper value for each input. Point 1), in particular, means having as many distinct net activation configurations as necessary to (at least) tell apart inputs needing (significantly, not in a statistical sense, just a qualitative sense) different outputs.
  I am inclined to think that if the input is complex enough, you would call what happens "compression".
  
  
  ---
  
  ### Answer by username_2 on 2016-09-08. Score: 1
  
  ANNs don't compress, they generalise.
  
  Often this leads to compression, i.e. the internal generalised representation is smaller than the original input, but not necessarily. Imagine a ANN that is trained to use the screen input of a computer game to play it. If the game is very rich and conceptually deep the internal representation of a single screen input might be a lot bigger than the input itself, because ANNs put the single data points into the context of the overall data. Which leads us to the second point:
  
  ANNs (and the neocortex) model data hierarchically.
  
  This is what makes them so powerful. So it is not just about having a large number of parameters, they also have to be arranged in such a way that they capture the structure of the data (or the world), which very often seems to be hierarchical. Just look a two different pictures of a duck. On the pixel level they might be as different as random images, all the similarities emerge in higher levels of the hierarchy, when enough pixels combined give you the patterns of feathers, beak and webs. Bloom filters obviously lack this property. They would only give you a pixel by pixel account of whether you have seen (almost) exactly this picture before.
  
  > **Comments:**
  > **username_0**: It looks like there is prior work in hierarchical data compression (see [here](http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.22.3180&rep=rep1&type=pdf)[pdf], [here](http://www.columbia.edu/~jw2966/papers/MDHW07-PAMI.pdf)[pdf] and [here](https://en.wikipedia.org/wiki/Set_partitioning_in_hierarchical_trees)).  
  
  ---
  
  