Answers.Body: <p>It doesn't drops rows or columns, it acts directly on scalars. The <a href="https://keras.io/api/layers/regularization_layers/dropout/" rel="nofollow noreferrer">Dropout Layer keras documentation</a> explains it and illustrates it with an example :</p>
  <blockquote>
  <p>The Dropout layer randomly sets input units to 0 with a frequency of rate</p>
  </blockquote>
  <p>After an Dense Layer, the Dropout inputs are directly the outputs of the Dense layer neurons, as you said. After your embedding layer, in your case, you should have <code>rate * (16 * input_length) = 0.2 * 20 * 16 = 64</code> inputs set to 0 out of the 320 scalars inputs. These 64 <em>dropped</em> inputs are randomly selected in the 20x16 grid. Note that the Dropout rescales the non dropped inputs by multiplicating them by a factor <span class="math-container">$\frac{1}{1-rate}$</span>.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-08-21T13:45:28.780Z
Answers.LastActivityDate: 2020-08-21T13:45:28.780Z
Answers.UserDisplayName: 16Aghnar
Body: <pre><code>model = tf.keras.Sequential([
      tf.keras.layers.Embedding(1000, 16, input_length=20), 
      tf.keras.layers.Dropout(0.2),                           # &lt;- How does the dropout work?
      tf.keras.layers.Conv1D(64, 5, activation='relu'),
      tf.keras.layers.MaxPooling1D(pool_size=4),
      tf.keras.layers.LSTM(64),
      tf.keras.layers.Dense(1, activation='sigmoid')
  ])
  </code></pre>
  <p>I can understand when dropout is applied between <code>Dense layers</code>, which randomly drops and prevents the former layer neurons from updating parameters. I don't understand how dropout works after an embedding layer.</p>
  <p>Let's say the output shape of the Embedding layer is <code>(batch_size,20,16)</code> or simply <code>(20,16)</code> if we ignore the batch size. How is dropout applied to the embedding layer's output?</p>
  <p>Randomly dropout rows or columns?</p>
  
Tags: <natural-language-processing><tensorflow><recurrent-neural-networks><long-short-term-memory><word-embedding>
Title: How is dropout applied to the embedding layer's output?
UserDisplayName: o_yeah
ThreadText: # How is dropout applied to the embedding layer's output?
  
  (Asked by: username_0 on 2020-08-21)
  
  ```
  model = tf.keras.Sequential([
      tf.keras.layers.Embedding(1000, 16, input_length=20), 
      tf.keras.layers.Dropout(0.2),                           # <- How does the dropout work?
      tf.keras.layers.Conv1D(64, 5, activation='relu'),
      tf.keras.layers.MaxPooling1D(pool_size=4),
      tf.keras.layers.LSTM(64),
      tf.keras.layers.Dense(1, activation='sigmoid')
  ])
  
  ```
  
  I can understand when dropout is applied between `Dense layers`, which randomly drops and prevents the former layer neurons from updating parameters. I don't understand how dropout works after an embedding layer.
  
  Let's say the output shape of the Embedding layer is `(batch_size,20,16)` or simply `(20,16)` if we ignore the batch size. How is dropout applied to the embedding layer's output?
  
  Randomly dropout rows or columns?
  
  **Tags:** natural-language-processing, tensorflow, recurrent-neural-networks, long-short-term-memory, word-embedding
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2020-08-21. Score: 1 | Accepted answer
  
  It doesn't drops rows or columns, it acts directly on scalars. The [Dropout Layer keras documentation](https://keras.io/api/layers/regularization_layers/dropout/) explains it and illustrates it with an example :
  
  > The Dropout layer randomly sets input units to 0 with a frequency of rate
  
  After an Dense Layer, the Dropout inputs are directly the outputs of the Dense layer neurons, as you said. After your embedding layer, in your case, you should have `rate * (16 * input_length) = 0.2 * 20 * 16 = 64` inputs set to 0 out of the 320 scalars inputs. These 64 *dropped* inputs are randomly selected in the 20x16 grid. Note that the Dropout rescales the non dropped inputs by multiplicating them by a factor $\frac{1}{1-rate}$.
  
  
  ---
  
  