Answers.Body: <p>Imagine, you want to re-compute the last layer of a pre-trained model : </p>
  
  <pre><code>Input-&gt;[Freezed-Layers]-&gt;[Last-Layer-To-Re-Compute]-&gt;Output
  </code></pre>
  
  <p>To train <em>[Last-Layer-To-Re-Compute]</em>, you need to evaluate outputs of <em>[Freezed-Layers]</em> multiple times for a given input data. In order to save time, you can compute these ouputs <strong>only once</strong>.</p>
  
  <pre><code>Input#1-&gt;[Freezed-Layers]-&gt;Bottleneck-Features-Of-Input#1
  </code></pre>
  
  <p>Then, you store all <em>Bottleneck-Features-Of-Input#i</em> and directly use them to train [Last-Layer-To-Re-Compute].</p>
  
  <p>Explanations from the "cache_bottlenecks" function of the "image_retraining" example : </p>
  
  <blockquote>
    <p>Because we're likely to read the same image multiple times (if there are no
        distortions applied during training) it can speed things up a lot if we
        calculate the bottleneck layer values once for each image during
        preprocessing, and then just read those cached values repeatedly during
        training.</p>
  </blockquote>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2018-03-19T17:18:06.333Z
Answers.LastActivityDate: 2018-03-20T13:15:52.760Z
Answers.LastEditDate: 2018-03-20T13:15:52.760Z
Answers.UserDisplayName: JC R
Answers.Body: <p>The bottleneck in a neural network is just a layer with fewer neurons than the layer below or above it. Having such a layer encourages the network to compress feature representations (of salient features for the target variable) to best fit in the available space. Improvements to compression occur due to the goal of reducing the cost function, as for all weight updates.</p>
  <p>In a CNN (such as Google's Inception network), bottleneck layers are added to reduce the number of feature maps (aka channels) in the network, which, otherwise, tend to increase in each layer. This is achieved by using 1x1 convolutions with fewer output channels than input channels.</p>
  <p>You don't usually calculate weights for bottleneck layers directly, the training process handles that, as for all other weights. Selecting a good size for a bottleneck layer is something you have to guess, and then experiment, in order to find network architectures that work well. The goal here is usually finding a network that generalises well to new images, and bottleneck layers help by reducing the number of parameters in the network whilst still allowing it to be deep and represent many feature maps.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-06-02T14:56:09.617Z
Answers.Text: Hi @NeilSlater, how is a bottleneck layer different from a bottleneck feature? I have read some kernels on Kaggle and here where people use a pretrained Resnet (after removing its last few layers). The model is made to generate the features without any further training. The features are then passed to a classifier of their choice, usually an SVM or a simpler CNN to learn from those features. Clearly in that case, the network will not learn to compress the feature representations in the available space.
Answers.UserDisplayName: Anshuman Kumar
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-06-02T14:59:29.700Z
Answers.Text: Like here the person uses ResNet https://www.kaggle.com/paperboiii/one-class-classification-for-images  or here https://towardsdatascience.com/building-the-hotdog-not-hotdog-classifier-from-hbos-silicon-valley-c0cb2317711f
Answers.UserDisplayName: Anshuman Kumar
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-06-02T15:00:28.110Z
Answers.Text: @NeilSlater, How much of a difference does compressing feature representations make to the overall accuracy?
Answers.UserDisplayName: Anshuman Kumar
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-06-02T15:16:21.710Z
Answers.Text: @AnshumanKumar: I suggest you ask a separate question about that on the site. In brief though, a feature vector (or embedding) that is good for one task is often good for a related task without re-training. It _might_ not be as good as if you trained from scratch, but the difference may not be important to you
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-01-05T23:38:44.793Z
Answers.Text: **in order to get the best loss during training**. How can a bottleneck layer be useful in getting the best loss during training?
Answers.UserDisplayName: hanugm
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-01-06T08:05:44.770Z
Answers.Text: @hanugm It is the other way around: The drive to get lower loss (through whatever optimisation is being implied) encourages higher fidellity compression, at least in regards to data from the original dataset that is meaningful in the context of whatever the network is predicting.
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-01-06T21:56:50.493Z
Answers.Text: It may be a good idea to comment on whether this terminology "bottlenecks" is standard. I've seen people use this word "bottleneck" to refer to many other similar concepts (e.g. as a synonym for "limitation") and probably I've also seen it being used in the context of neural networks, but I am not sure if this is standard terminology, so you may want to note that in your answer. For example, if the authors of the Inception papers use this term, it may be a good idea to provide a link to the paper(s).
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-01-06T22:07:46.963Z
Answers.Text: "_...This penultimate layer has been trained to output a set of values that's good enough for the classifier to use to distinguish between all the classes it's been asked to recognize._". So, it doesn't say that this layer has fewer neurons. That may not be the case or it might be the case. I didn't fully read the article.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-01-06T22:12:50.343Z
Answers.Text: If you think of architectures like u-net, then your usage/definition of the term "bottleneck" would be different than the definition provided in the article the OP was referring to. The u-net is an auto-encoder-like architecture where the layer with fewer neurons is not the layer just before the output layer.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2017-12-31T09:41:25.500Z
Answers.LastActivityDate: 2022-01-06T08:11:02.587Z
Answers.LastEditDate: 2022-01-06T08:11:02.587Z
Answers.UserDisplayName: Neil Slater
Body: <p>What are &quot;bottlenecks&quot; in the context of neural networks?</p>
  <p>This term is mentioned, for example, in <a href="https://web.archive.org/web/20180703133602/https://www.tensorflow.org/tutorials/image_retraining" rel="noreferrer">this TensorFlow article</a>, which also uses the term &quot;bottleneck values&quot;. How does one calculate bottleneck values? How do these values help image classification?</p>
  <p>Please explain in simple words.</p>
  
Tags: <neural-networks><deep-learning><tensorflow><terminology><bottlenecks>
Title: What are "bottlenecks" in neural networks?
UserDisplayName: Anurag Singh
ThreadText: # What are "bottlenecks" in neural networks?
  
  (Asked by: username_0 on 2017-12-28)
  
  What are "bottlenecks" in the context of neural networks?
  
  This term is mentioned, for example, in [this TensorFlow article](https://web.archive.org/web/20180703133602/https://www.tensorflow.org/tutorials/image_retraining), which also uses the term "bottleneck values". How does one calculate bottleneck values? How do these values help image classification?
  
  Please explain in simple words.
  
  **Tags:** neural-networks, deep-learning, tensorflow, terminology, bottlenecks
  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_2 on 2017-12-31. Score: 39 | Accepted answer
  
  The bottleneck in a neural network is just a layer with fewer neurons than the layer below or above it. Having such a layer encourages the network to compress feature representations (of salient features for the target variable) to best fit in the available space. Improvements to compression occur due to the goal of reducing the cost function, as for all weight updates.
  
  In a CNN (such as Google's Inception network), bottleneck layers are added to reduce the number of feature maps (aka channels) in the network, which, otherwise, tend to increase in each layer. This is achieved by using 1x1 convolutions with fewer output channels than input channels.
  
  You don't usually calculate weights for bottleneck layers directly, the training process handles that, as for all other weights. Selecting a good size for a bottleneck layer is something you have to guess, and then experiment, in order to find network architectures that work well. The goal here is usually finding a network that generalises well to new images, and bottleneck layers help by reducing the number of parameters in the network whilst still allowing it to be deep and represent many feature maps.
  
  > **Comments:**
  > **username_3**: Hi @username_2, how is a bottleneck layer different from a bottleneck feature? I have read some kernels on Kaggle and here where people use a pretrained Resnet (after removing its last few layers). The model is made to generate the features without any further training. The features are then passed to a classifier of their choice, usually an SVM or a simpler CNN to learn from those features. Clearly in that case, the network will not learn to compress the feature representations in the available space.  
  > **username_3**: Like here the person uses ResNet https://www.kaggle.com/paperboiii/one-class-classification-for-images  or here https://towardsdatascience.com/building-the-hotdog-not-hotdog-classifier-from-hbos-silicon-valley-c0cb2317711f  
  > **username_3**: @username_2, How much of a difference does compressing feature representations make to the overall accuracy?  
  > **username_2**: @username_3: I suggest you ask a separate question about that on the site. In brief though, a feature vector (or embedding) that is good for one task is often good for a related task without re-training. It _might_ not be as good as if you trained from scratch, but the difference may not be important to you  
  > **username_4**: **in order to get the best loss during training**. How can a bottleneck layer be useful in getting the best loss during training?  
  > **username_2**: @username_4 It is the other way around: The drive to get lower loss (through whatever optimisation is being implied) encourages higher fidellity compression, at least in regards to data from the original dataset that is meaningful in the context of whatever the network is predicting.  
  > **username_5**: It may be a good idea to comment on whether this terminology "bottlenecks" is standard. I've seen people use this word "bottleneck" to refer to many other similar concepts (e.g. as a synonym for "limitation") and probably I've also seen it being used in the context of neural networks, but I am not sure if this is standard terminology, so you may want to note that in your answer. For example, if the authors of the Inception papers use this term, it may be a good idea to provide a link to the paper(s).  
  > **username_5**: "_...This penultimate layer has been trained to output a set of values that's good enough for the classifier to use to distinguish between all the classes it's been asked to recognize._". So, it doesn't say that this layer has fewer neurons. That may not be the case or it might be the case. I didn't fully read the article.  
  > **username_5**: If you think of architectures like u-net, then your usage/definition of the term "bottleneck" would be different than the definition provided in the article the OP was referring to. The u-net is an auto-encoder-like architecture where the layer with fewer neurons is not the layer just before the output layer.  
  
  ---
  
  ### Answer by username_1 on 2018-03-19. Score: 11
  
  Imagine, you want to re-compute the last layer of a pre-trained model :
  
  ```
  Input->[Freezed-Layers]->[Last-Layer-To-Re-Compute]->Output
  
  ```
  
  To train *[Last-Layer-To-Re-Compute]*, you need to evaluate outputs of *[Freezed-Layers]* multiple times for a given input data. In order to save time, you can compute these ouputs **only once**.
  
  ```
  Input#1->[Freezed-Layers]->Bottleneck-Features-Of-Input#1
  
  ```
  
  Then, you store all *Bottleneck-Features-Of-Input#i* and directly use them to train [Last-Layer-To-Re-Compute].
  
  Explanations from the "cache\_bottlenecks" function of the "image\_retraining" example :
  
  > Because we're likely to read the same image multiple times (if there are no
  > distortions applied during training) it can speed things up a lot if we
  > calculate the bottleneck layer values once for each image during
  > preprocessing, and then just read those cached values repeatedly during
  > training.
  
  
  ---
  
  