Answers.Body: <blockquote>
  <p><strong>A bottleneck layer is a layer that contains few nodes compared to the previous layers. It can be used to obtain a representation of the
  input with reduced dimensionality.</strong></p>
  </blockquote>
  <p>If you create bottlenecks in the initial layers it can cause loss of information.</p>
  <p><strong>Why it is done?</strong></p>
  <p>Take the example of an image dataset that contains high-resolution images. High resolution means more pixels means it will need more nodes in the input layer.
  Having more nodes will need more computational power to train the network. Hence in such cases, we can use fewer nodes in the next layer. As images are high resolution we might not lose any important information.</p>
  <p><a href="https://i.sstatic.net/WIZ2w.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/WIZ2w.png" alt="Feedforward Network with bottleneck" /></a></p>
  <p><a href="https://i.sstatic.net/LE4iy.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/LE4iy.png" alt="Bottleneck" /></a></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-09-09T13:08:12.753Z
Answers.Text: You should at least acknowledge the source from which you took that excerpt, which seems to be [this answer](https://stats.stackexchange.com/a/262060). So, please, edit your post to provide a link to that answer or the source where you took that quote from. In general, you should not just copy and paste content from places without citing, as plagiarism is at least discouraged and at most not allowed here. Having said that, your answer seems to be partially inconsistent with the quote that the asker provided because it states that the dimensionality is not sufficient for "information content".
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-09-01T04:21:34.697Z
Answers.LastActivityDate: 2021-09-01T07:46:38.487Z
Answers.LastEditDate: 2021-09-01T07:46:38.487Z
Answers.UserDisplayName: Harshal
Body: <p>Consider the following paragraph from <code>section 2: General Design Principles</code> of the research paper titled <a href="https://www.cv-foundation.org/openaccess/content_cvpr_2016/papers/Szegedy_Rethinking_the_Inception_CVPR_2016_paper.pdf" rel="nofollow noreferrer">Rethinking the Inception Architecture for Computer Vision</a></p>
  <blockquote>
  <p>Avoid <strong>representational bottlenecks</strong>, especially early in the
  network. Feed-forward networks can be represented by an acyclic graph
  from the input layer(s) to the classifier or regressor. This defines a
  clear direction for the information flow. For any cut separating the
  inputs from the outputs, one can access the amount of information
  passing though the cut. One should avoid <strong>bottlenecks</strong> with extreme
  compression. In general the representation size should gently decrease
  from the inputs to the outputs before reaching the final
  representation used for the task at hand. Theoretically, information
  content cannot be assessed merely by the dimensionality of the
  representation as it discards important factors like correlation
  structure; the dimensionality merely provides a rough estimate of
  information content.</p>
  </blockquote>
  <p>This paragraph warned us to avoid bottlenecks and also representational bottlenecks. What does it mean by a bottleneck of/in a neural network and representational bottleneck?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-09-09T13:14:21.423Z
Comments.Text: I don't feel like providing a formal answer because I am not sure whether these terms "bottlenecks" are formally defined. In fact, we had similar questions in the past,  [here](https://ai.stackexchange.com/q/3089/2444) or [this](https://ai.stackexchange.com/q/4864/2444) (actually, I think your question is a duplicate of this last one), but it seems to me that these terms have been used more loosely. Having said that, I think the authors of that quote are using the term "representational bottleneck" as a synonym for just "bottleneck".
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-01-05T23:43:11.293Z
Comments.Text: Yeah, @nbro. After understanding the terminology regarding this, I also feel that this is a duplicate one. But, while asking the question, I thought that the word bottleneck may be used for different purposes and hence I asked.
Comments.UserDisplayName: hanugm
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-10-28T03:29:15.827Z
Comments.Text: [what-are-bottlenecks-in-neural-networks?](https://ai.stackexchange.com/questions/4864/what-are-bottlenecks-in-neural-networks?noredirect=1&lq=1)
Comments.UserDisplayName: Anurag Singh
Tags: <neural-networks><terminology><bottlenecks>
Title: What does it mean by bottleneck and representational bottleneck in feedforward neural networks?
UserDisplayName: hanugm
ThreadText: # What does it mean by bottleneck and representational bottleneck in feedforward neural networks?
  
  (Asked by: username_0 on 2021-09-01)
  
  Consider the following paragraph from `section 2: General Design Principles` of the research paper titled [Rethinking the Inception Architecture for Computer Vision](https://www.cv-foundation.org/openaccess/content_cvpr_2016/papers/Szegedy_Rethinking_the_Inception_CVPR_2016_paper.pdf)
  
  > Avoid **representational bottlenecks**, especially early in the
  > network. Feed-forward networks can be represented by an acyclic graph
  > from the input layer(s) to the classifier or regressor. This defines a
  > clear direction for the information flow. For any cut separating the
  > inputs from the outputs, one can access the amount of information
  > passing though the cut. One should avoid **bottlenecks** with extreme
  > compression. In general the representation size should gently decrease
  > from the inputs to the outputs before reaching the final
  > representation used for the task at hand. Theoretically, information
  > content cannot be assessed merely by the dimensionality of the
  > representation as it discards important factors like correlation
  > structure; the dimensionality merely provides a rough estimate of
  > information content.
  
  This paragraph warned us to avoid bottlenecks and also representational bottlenecks. What does it mean by a bottleneck of/in a neural network and representational bottleneck?
  
  **Tags:** neural-networks, terminology, bottlenecks
  
  > **Comments:**
  > **username_1**: I don't feel like providing a formal answer because I am not sure whether these terms "bottlenecks" are formally defined. In fact, we had similar questions in the past,  [here](https://ai.stackexchange.com/q/3089/2444) or [this](https://ai.stackexchange.com/q/4864/2444) (actually, I think your question is a duplicate of this last one), but it seems to me that these terms have been used more loosely. Having said that, I think the authors of that quote are using the term "representational bottleneck" as a synonym for just "bottleneck".  
  > **username_0**: Yeah, @username_1. After understanding the terminology regarding this, I also feel that this is a duplicate one. But, while asking the question, I thought that the word bottleneck may be used for different purposes and hence I asked.  
  > **username_2**: [what-are-bottlenecks-in-neural-networks?](https://ai.stackexchange.com/questions/4864/what-are-bottlenecks-in-neural-networks?noredirect=1&lq=1)  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_3 on 2021-09-01. Score: 1
  
  > **A bottleneck layer is a layer that contains few nodes compared to the previous layers. It can be used to obtain a representation of the
  > input with reduced dimensionality.**
  
  If you create bottlenecks in the initial layers it can cause loss of information.
  
  **Why it is done?**
  
  Take the example of an image dataset that contains high-resolution images. High resolution means more pixels means it will need more nodes in the input layer.
  Having more nodes will need more computational power to train the network. Hence in such cases, we can use fewer nodes in the next layer. As images are high resolution we might not lose any important information.
  
  [![Feedforward Network with bottleneck](https://i.sstatic.net/WIZ2w.png)](https://i.sstatic.net/WIZ2w.png)
  
  [![Bottleneck](https://i.sstatic.net/LE4iy.png)](https://i.sstatic.net/LE4iy.png)
  
  > **Comments:**
  > **username_1**: You should at least acknowledge the source from which you took that excerpt, which seems to be [this answer](https://stats.stackexchange.com/a/262060). So, please, edit your post to provide a link to that answer or the source where you took that quote from. In general, you should not just copy and paste content from places without citing, as plagiarism is at least discouraged and at most not allowed here. Having said that, your answer seems to be partially inconsistent with the quote that the asker provided because it states that the dimensionality is not sufficient for "information content".  
  
  ---
  
  