Answers.Body: <p>I'm going to post another guess to this question - it won't be a complete answer, but hopefully it'll provide some direction towards finding a more legitimate answer.</p>
  
  <p>The feed-forward networks as suggested by Vaswani are very reminiscent of the <a href="https://web.stanford.edu/class/cs294a/sparseAutoencoder.pdf" rel="nofollow noreferrer">sparse autoencoders</a>. Where the input / output dimensions are much greater than the hidden input dimension.</p>
  
  <p><a href="https://i.sstatic.net/jDlzn.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/jDlzn.png" alt="enter image description here"></a></p>
  
  <p>If you aren't familiar with sparse autoencoders, this is a little counter intuitive - WTF would you have a larger hidden dimension?  </p>
  
  <p>The intuition borrows from infinitely wide neural networks.  If you have an infinitely wide neural network, you have basically have a Gaussian process and sample any function you'd like.  So the wider the network you have, the more approximation power that you have.  In the case of inputs, this is a matter of learning a dictionary. If you have only discrete inputs, this hidden layer will be capped at <span class="math-container">$O(2^N)$</span> width, where <span class="math-container">$N$</span> is the maximum number of bits it takes to represent the input (which would boil down to approximating a lookup table). </p>
  
  <p>Of course, these aren't trivial to implement in practice.  These layers are bound to be bloated with identifiability issues.  Common approaches include <span class="math-container">$L_1$</span> regularization.  I'm guessing that the convolutional layers + dropout are just another attempt to deal with these sorts of identifiability issues.  Furthermore, the FFN is an attempt to learn an arbitrary mapping for individual words (you can think of mapping words to synonyms for instance).</p>
  
  <p>These are all guesses though - more intuition is welcome.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-11-27T07:52:15.557Z
Answers.Text: "Where the input / output dimensions are much greater than the hidden input dimension." guess this is a typo? From your content it sounds like the number of hidden dimensions is much larger than the input/output dimensions instead.
Answers.UserDisplayName: chichi
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-02-12T20:30:20.947Z
Answers.LastActivityDate: 2020-02-12T20:30:20.947Z
Answers.UserDisplayName: mortonjt
Answers.Body: <p>1) The math is the exact same, so from an optimization or mathematical perspective there is no difference  </p>
  
  <p>2) Here are my <strong>guesses</strong> to a possible answer.  </p>
  
  <ul>
  <li>Habit: People may just call one over the other out of habit</li>
  <li>Generality: Across frameworks a 1d convolution op would work, while Dense of FC may need adjustments to work on the temporal axis </li>
  <li>Parallel Workers: Convolution and Dense call different subroutines in the backend, and the one used by convolution may have better gains on sequential input for this purpose </li>
  </ul>
  
  <p><strong>Edit</strong><br>
  Regarding bench-marking the 2, your experiment was shallow. I didn't have time to wait to do a full gird search, so i held 3 paramaters constant and fluctuated one. Here are the results (note the model was just a simple feed forward relu residual model) </p>
  
  <p><a href="https://i.sstatic.net/Khm6S.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/Khm6S.png" alt="enter image description here"></a>
  <a href="https://i.sstatic.net/xl5b0.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/xl5b0.png" alt="enter image description here"></a> 
  <a href="https://i.sstatic.net/O7XXV.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/O7XXV.png" alt="enter image description here"></a>
  <a href="https://i.sstatic.net/VK07u.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/VK07u.png" alt="enter image description here"></a></p>
  
  <p>Note that in a couple yeah dense out performs conv but it isn't consistent and there are scenarios where it is not true. This is only for a small grid that I chose but you can extend this yourself to check. So it is not as straightforward to say one is sheerly better than the other. </p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-02-07T15:13:38.550Z
Answers.Text: I have benchmarked performance on various input sizes and a simple Dense implementation seems to win all the time with respect to speed. It's very hard to benchmark memory consumption in Tensorflow, because it is not exactly deterministic, but I don't see any possible optimisations here either. By the way, your sidenote has already been written in the question.
Answers.UserDisplayName: Eli Korvigo
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-02-07T15:54:35.837Z
Answers.Text: @Eli 1) Hmm I cant imagine that being true for large models. What models were the benchmarks on. 2) whoops, i gotta read more lol
Answers.UserDisplayName: mshlis
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-02-12T20:14:36.980Z
Answers.Text: I've added some benchmarks to my question
Answers.UserDisplayName: Eli Korvigo
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-02-12T21:30:48.757Z
Answers.Text: @EliKorvigo I just ran a couple benchmarks that extend past just a single run
Answers.UserDisplayName: mshlis
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-02-07T14:38:29.700Z
Answers.LastActivityDate: 2020-02-12T21:30:22.093Z
Answers.LastEditDate: 2020-02-12T21:30:22.093Z
Answers.UserDisplayName: mshlis
Body: <p>The Transformer model introduced in <a href="https://arxiv.org/abs/1706.03762" rel="noreferrer">"Attention is all you need"</a> by Vaswani et al. incorporates a so-called position-wise feed-forward network (FFN):</p>
  
  <blockquote>
    <p>In addition to attention sub-layers, each of the layers in our encoder
    and decoder contains a fully connected feed-forward network, which is
    applied to each position separately and identically. This consists of
    two linear transformations with a ReLU activation in between.</p>
    
    <p><span class="math-container">$$\text{FFN}(x) = \max(0, x \times {W}_{1} + {b}_{1}) \times {W}_{2} + {b}_{2}$$</span></p>
    
    <p>While the linear transformations are the same across different positions, they use different parameters from layer to layer. Another way of describing this is as two convolutions with kernel size 1. The dimensionality of input and output is <span class="math-container">${d}_{\text{model}} = 512$</span>, and the inner-layer has dimensionality <span class="math-container">${d}_{ff} = 2048$</span>.</p>
  </blockquote>
  
  <p>I have seen at least one implementation in Keras that directly follows the convolution analogy. Here is an excerpt from <a href="https://github.com/Lsdefine/attention-is-all-you-need-keras/blob/master/transformer.py" rel="noreferrer">attention-is-all-you-need-keras</a>.</p>
  
  <pre><code>class PositionwiseFeedForward():
      def __init__(self, d_hid, d_inner_hid, dropout=0.1):
          self.w_1 = Conv1D(d_inner_hid, 1, activation='relu')
          self.w_2 = Conv1D(d_hid, 1)
          self.layer_norm = LayerNormalization()
          self.dropout = Dropout(dropout)
      def __call__(self, x):
          output = self.w_1(x) 
          output = self.w_2(output)
          output = self.dropout(output)
          output = Add()([output, x])
          return self.layer_norm(output)
  </code></pre>
  
  <p>Yet, in Keras you can apply a single <code>Dense</code> layer across all time-steps using the <code>TimeDistributed</code> wrapper (moreover, a simple <code>Dense</code> layer applied to a 2D input <a href="https://stackoverflow.com/a/44616780/3846213">implicitly behaves</a> like a <code>TimeDistributed</code> layer). Therefore, in Keras a stack of two Dense layers (one with a ReLU and the other one without an activation) is exactly the same thing as the aforementioned position-wise FFN. So, why would you implement it using convolutions?</p>
  
  <p><strong>Update</strong> </p>
  
  <p>Adding benchmarks in response to the answer by @mshlis:</p>
  
  <pre><code>import os
  import typing as t
  os.environ['CUDA_VISIBLE_DEVICES'] = '0'
  
  import numpy as np
  
  from keras import layers, models
  from keras import backend as K
  from tensorflow import Tensor
  
  
  # Generate random data
  
  n = 128000  # n samples
  seq_l = 32  # sequence length
  emb_dim = 512  # embedding size
  
  x = np.random.normal(0, 1, size=(n, seq_l, emb_dim)).astype(np.float32)
  y = np.random.binomial(1, 0.5, size=n).astype(np.int32)
  </code></pre>
  
  <hr>
  
  <pre><code># Define constructors
  
  def ffn_dense(hid_dim: int, input_: Tensor) -&gt; Tensor:
      output_dim = K.int_shape(input_)[-1]
      hidden = layers.Dense(hid_dim, activation='relu')(input_)
      return layers.Dense(output_dim, activation=None)(hidden)
  
  
  def ffn_cnn(hid_dim: int, input_: Tensor) -&gt; Tensor:
      output_dim = K.int_shape(input_)[-1]
      hidden = layers.Conv1D(hid_dim, 1, activation='relu')(input_)
      return layers.Conv1D(output_dim, 1, activation=None)(hidden)
  
  
  def build_model(ffn_implementation: t.Callable[[int, Tensor], Tensor], 
                  ffn_hid_dim: int, 
                  input_shape: t.Tuple[int, int]) -&gt; models.Model:
      input_ = layers.Input(shape=(seq_l, emb_dim))
      ffn = ffn_implementation(ffn_hid_dim, input_)
      flattened = layers.Flatten()(ffn)
      output = layers.Dense(1, activation='sigmoid')(flattened)
      model = models.Model(inputs=input_, outputs=output)
      model.compile(optimizer='Adam', loss='binary_crossentropy')
      return model
  </code></pre>
  
  <hr>
  
  <pre><code># Build the models
  
  ffn_hid_dim = emb_dim * 4  # this rule is taken from the original paper
  bath_size = 512  # the batchsize was selected to maximise GPU load, i.e. reduce PCI IO overhead
  
  model_dense = build_model(ffn_dense, ffn_hid_dim, (seq_l, emb_dim))
  model_cnn = build_model(ffn_cnn, ffn_hid_dim, (seq_l, emb_dim))
  </code></pre>
  
  <hr>
  
  <pre><code># Pre-heat the GPU and let TF apply memory stream optimisations
  
  model_dense.fit(x=x, y=y[:, None], batch_size=bath_size, epochs=1)
  %timeit model_dense.fit(x=x, y=y[:, None], batch_size=bath_size, epochs=1)
  
  model_cnn.fit(x=x, y=y[:, None], batch_size=bath_size, epochs=1)
  %timeit model_cnn.fit(x=x, y=y[:, None], batch_size=bath_size, epochs=1)
  </code></pre>
  
  <p>I am getting 14.8 seconds per epoch with the Dense implementation:</p>
  
  <pre><code>Epoch 1/1
  128000/128000 [==============================] - 15s 116us/step - loss: 0.6332
  Epoch 1/1
  128000/128000 [==============================] - 15s 115us/step - loss: 0.5327
  Epoch 1/1
  128000/128000 [==============================] - 15s 117us/step - loss: 0.3828
  Epoch 1/1
  128000/128000 [==============================] - 14s 113us/step - loss: 0.2543
  Epoch 1/1
  128000/128000 [==============================] - 15s 116us/step - loss: 0.1908
  Epoch 1/1
  128000/128000 [==============================] - 15s 116us/step - loss: 0.1533
  Epoch 1/1
  128000/128000 [==============================] - 15s 117us/step - loss: 0.1475
  Epoch 1/1
  128000/128000 [==============================] - 15s 117us/step - loss: 0.1406
  
  14.8 s ± 170 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
  </code></pre>
  
  <p>and 18.2 seconds for the CNN implementation. I am running this test on a standard Nvidia RTX 2080.
  So, from a performance perspective there seems to be no point in actually implementing an FFN block as a CNN in Keras. Considering that the maths are the same, the choice boils down to pure aesthetics. </p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-02-05T22:05:01.190Z
Comments.Text: There is a similar question here on Position-wise FFNs (see here: https://datascience.stackexchange.com/questions/55647/multi-head-attention-mechanism-in-transformer-and-need-of-feed-forward-neural-ne/55722).  However, I'm not convinced by this answer.
Comments.UserDisplayName: mortonjt
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-02-12T20:21:55.207Z
Comments.Text: @mortonjt Hello, Jamie. I am not sure the questions are all that similar, though they are related. I've added some benchmarks to my question: at this point I'm almost convinced that the CNN-interpretation/implementation is there, because the authors like the analogy (and/or someone might find it easier to interpret the operation that way). P.S. it's nice seeing you.
Comments.UserDisplayName: Eli Korvigo
Tags: <deep-learning><keras><convolution><transformer><feedforward-neural-networks>
Title: Why would you implement the position-wise feed-forward network of the transformer with convolution layers?
UserDisplayName: Eli Korvigo
ThreadText: # Why would you implement the position-wise feed-forward network of the transformer with convolution layers?
  
  (Asked by: username_0 on 2019-09-18)
  
  The Transformer model introduced in ["Attention is all you need"](https://arxiv.org/abs/1706.03762) by Vaswani et al. incorporates a so-called position-wise feed-forward network (FFN):
  
  > In addition to attention sub-layers, each of the layers in our encoder
  > and decoder contains a fully connected feed-forward network, which is
  > applied to each position separately and identically. This consists of
  > two linear transformations with a ReLU activation in between.
  >
  > $$\text{FFN}(x) = \max(0, x \times {W}\_{1} + {b}\_{1}) \times {W}\_{2} + {b}\_{2}$$
  >
  > While the linear transformations are the same across different positions, they use different parameters from layer to layer. Another way of describing this is as two convolutions with kernel size 1. The dimensionality of input and output is ${d}\_{\text{model}} = 512$, and the inner-layer has dimensionality ${d}\_{ff} = 2048$.
  
  I have seen at least one implementation in Keras that directly follows the convolution analogy. Here is an excerpt from [attention-is-all-you-need-keras](https://github.com/Lsdefine/attention-is-all-you-need-keras/blob/master/transformer.py).
  
  ```
  class PositionwiseFeedForward():
      def __init__(self, d_hid, d_inner_hid, dropout=0.1):
          self.w_1 = Conv1D(d_inner_hid, 1, activation='relu')
          self.w_2 = Conv1D(d_hid, 1)
          self.layer_norm = LayerNormalization()
          self.dropout = Dropout(dropout)
      def __call__(self, x):
          output = self.w_1(x) 
          output = self.w_2(output)
          output = self.dropout(output)
          output = Add()([output, x])
          return self.layer_norm(output)
  
  ```
  
  Yet, in Keras you can apply a single `Dense` layer across all time-steps using the `TimeDistributed` wrapper (moreover, a simple `Dense` layer applied to a 2D input [implicitly behaves](https://stackoverflow.com/a/44616780/3846213) like a `TimeDistributed` layer). Therefore, in Keras a stack of two Dense layers (one with a ReLU and the other one without an activation) is exactly the same thing as the aforementioned position-wise FFN. So, why would you implement it using convolutions?
  
  **Update**
  
  Adding benchmarks in response to the answer by @username_3:
  
  ```
  import os
  import typing as t
  os.environ['CUDA_VISIBLE_DEVICES'] = '0'
  
  import numpy as np
  
  from keras import layers, models
  from keras import backend as K
  from tensorflow import Tensor
  
  # Generate random data
  
  n = 128000  # n samples
  seq_l = 32  # sequence length
  emb_dim = 512  # embedding size
  
  x = np.random.normal(0, 1, size=(n, seq_l, emb_dim)).astype(np.float32)
  y = np.random.binomial(1, 0.5, size=n).astype(np.int32)
  
  ```
  
  ---
  
  ```
  # Define constructors
  
  def ffn_dense(hid_dim: int, input_: Tensor) -> Tensor:
      output_dim = K.int_shape(input_)[-1]
      hidden = layers.Dense(hid_dim, activation='relu')(input_)
      return layers.Dense(output_dim, activation=None)(hidden)
  
  def ffn_cnn(hid_dim: int, input_: Tensor) -> Tensor:
      output_dim = K.int_shape(input_)[-1]
      hidden = layers.Conv1D(hid_dim, 1, activation='relu')(input_)
      return layers.Conv1D(output_dim, 1, activation=None)(hidden)
  
  def build_model(ffn_implementation: t.Callable[[int, Tensor], Tensor], 
                  ffn_hid_dim: int, 
                  input_shape: t.Tuple[int, int]) -> models.Model:
      input_ = layers.Input(shape=(seq_l, emb_dim))
      ffn = ffn_implementation(ffn_hid_dim, input_)
      flattened = layers.Flatten()(ffn)
      output = layers.Dense(1, activation='sigmoid')(flattened)
      model = models.Model(inputs=input_, outputs=output)
      model.compile(optimizer='Adam', loss='binary_crossentropy')
      return model
  
  ```
  
  ---
  
  ```
  # Build the models
  
  ffn_hid_dim = emb_dim * 4  # this rule is taken from the original paper
  bath_size = 512  # the batchsize was selected to maximise GPU load, i.e. reduce PCI IO overhead
  
  model_dense = build_model(ffn_dense, ffn_hid_dim, (seq_l, emb_dim))
  model_cnn = build_model(ffn_cnn, ffn_hid_dim, (seq_l, emb_dim))
  
  ```
  
  ---
  
  ```
  # Pre-heat the GPU and let TF apply memory stream optimisations
  
  model_dense.fit(x=x, y=y[:, None], batch_size=bath_size, epochs=1)
  %timeit model_dense.fit(x=x, y=y[:, None], batch_size=bath_size, epochs=1)
  
  model_cnn.fit(x=x, y=y[:, None], batch_size=bath_size, epochs=1)
  %timeit model_cnn.fit(x=x, y=y[:, None], batch_size=bath_size, epochs=1)
  
  ```
  
  I am getting 14.8 seconds per epoch with the Dense implementation:
  
  ```
  Epoch 1/1
  128000/128000 [==============================] - 15s 116us/step - loss: 0.6332
  Epoch 1/1
  128000/128000 [==============================] - 15s 115us/step - loss: 0.5327
  Epoch 1/1
  128000/128000 [==============================] - 15s 117us/step - loss: 0.3828
  Epoch 1/1
  128000/128000 [==============================] - 14s 113us/step - loss: 0.2543
  Epoch 1/1
  128000/128000 [==============================] - 15s 116us/step - loss: 0.1908
  Epoch 1/1
  128000/128000 [==============================] - 15s 116us/step - loss: 0.1533
  Epoch 1/1
  128000/128000 [==============================] - 15s 117us/step - loss: 0.1475
  Epoch 1/1
  128000/128000 [==============================] - 15s 117us/step - loss: 0.1406
  
  14.8 s ± 170 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
  
  ```
  
  and 18.2 seconds for the CNN implementation. I am running this test on a standard Nvidia RTX 2080.
  So, from a performance perspective there seems to be no point in actually implementing an FFN block as a CNN in Keras. Considering that the maths are the same, the choice boils down to pure aesthetics.
  
  **Tags:** deep-learning, keras, convolution, transformer, feedforward-neural-networks
  
  > **Comments:**
  > **username_1**: There is a similar question here on Position-wise FFNs (see here: https://datascience.stackexchange.com/questions/55647/multi-head-attention-mechanism-in-transformer-and-need-of-feed-forward-neural-ne/55722).  However, I'm not convinced by this answer.  
  > **username_0**: @username_1 Hello, Jamie. I am not sure the questions are all that similar, though they are related. I've added some benchmarks to my question: at this point I'm almost convinced that the CNN-interpretation/implementation is there, because the authors like the analogy (and/or someone might find it easier to interpret the operation that way). P.S. it's nice seeing you.  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_1 on 2020-02-12. Score: 4
  
  I'm going to post another guess to this question - it won't be a complete answer, but hopefully it'll provide some direction towards finding a more legitimate answer.
  
  The feed-forward networks as suggested by Vaswani are very reminiscent of the [sparse autoencoders](https://web.stanford.edu/class/cs294a/sparseAutoencoder.pdf). Where the input / output dimensions are much greater than the hidden input dimension.
  
  [![enter image description here](https://i.sstatic.net/jDlzn.png)](https://i.sstatic.net/jDlzn.png)
  
  If you aren't familiar with sparse autoencoders, this is a little counter intuitive - WTF would you have a larger hidden dimension?
  
  The intuition borrows from infinitely wide neural networks. If you have an infinitely wide neural network, you have basically have a Gaussian process and sample any function you'd like. So the wider the network you have, the more approximation power that you have. In the case of inputs, this is a matter of learning a dictionary. If you have only discrete inputs, this hidden layer will be capped at $O(2^N)$ width, where $N$ is the maximum number of bits it takes to represent the input (which would boil down to approximating a lookup table).
  
  Of course, these aren't trivial to implement in practice. These layers are bound to be bloated with identifiability issues. Common approaches include $L\_1$ regularization. I'm guessing that the convolutional layers + dropout are just another attempt to deal with these sorts of identifiability issues. Furthermore, the FFN is an attempt to learn an arbitrary mapping for individual words (you can think of mapping words to synonyms for instance).
  
  These are all guesses though - more intuition is welcome.
  
  > **Comments:**
  > **username_2**: "Where the input / output dimensions are much greater than the hidden input dimension." guess this is a typo? From your content it sounds like the number of hidden dimensions is much larger than the input/output dimensions instead.  
  
  ---
  
  ### Answer by username_3 on 2020-02-07. Score: 3
  
  1) The math is the exact same, so from an optimization or mathematical perspective there is no difference
  
  2) Here are my **guesses** to a possible answer.
  
  * Habit: People may just call one over the other out of habit
  * Generality: Across frameworks a 1d convolution op would work, while Dense of FC may need adjustments to work on the temporal axis
  * Parallel Workers: Convolution and Dense call different subroutines in the backend, and the one used by convolution may have better gains on sequential input for this purpose
  
  **Edit**  
  Regarding bench-marking the 2, your experiment was shallow. I didn't have time to wait to do a full gird search, so i held 3 paramaters constant and fluctuated one. Here are the results (note the model was just a simple feed forward relu residual model)
  
  [![enter image description here](https://i.sstatic.net/Khm6S.png)](https://i.sstatic.net/Khm6S.png)
  [![enter image description here](https://i.sstatic.net/xl5b0.png)](https://i.sstatic.net/xl5b0.png)
  [![enter image description here](https://i.sstatic.net/O7XXV.png)](https://i.sstatic.net/O7XXV.png)
  [![enter image description here](https://i.sstatic.net/VK07u.png)](https://i.sstatic.net/VK07u.png)
  
  Note that in a couple yeah dense out performs conv but it isn't consistent and there are scenarios where it is not true. This is only for a small grid that I chose but you can extend this yourself to check. So it is not as straightforward to say one is sheerly better than the other.
  
  > **Comments:**
  > **username_0**: I have benchmarked performance on various input sizes and a simple Dense implementation seems to win all the time with respect to speed. It's very hard to benchmark memory consumption in Tensorflow, because it is not exactly deterministic, but I don't see any possible optimisations here either. By the way, your sidenote has already been written in the question.  
  > **username_3**: @Eli 1) Hmm I cant imagine that being true for large models. What models were the benchmarks on. 2) whoops, i gotta read more lol  
  > **username_0**: I've added some benchmarks to my question  
  > **username_3**: @username_0 I just ran a couple benchmarks that extend past just a single run  
  
  ---
  
  