Answers.Body: <p>I can't provide you with numbers and results, but I'd expect (for not triavial problems) the MLP-based model to be worse than a Transformer.</p>
  <p>The reason is that transformers are designed to handle sequences, whereas MLP are not (even if you flatten the entire sequence into a vector). Trasformers can leverage an <strong>inductive bias</strong> specifically designed for sequences (likewise CNNs for images), which should account for a better learning, at least, even in case of similar number of parameters and regularization.</p>
  <p>Thanks to <em>self-attention</em> transformers are able to learn to focus on relevant parts of the sequence, even far in time, whereas simple MLPs cannot do this. Moreover, <em>positional encoding</em> provides the transformer to use the sequence order information, enabling it to learn to attend even to relative positions. Instead, MLPs just combine information from all the vectors' components.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-06-04T08:35:38.530Z
Answers.Text: Not to nitpick, but vector components of course have ordering (it's a list, not a set, in python terminology)
Answers.UserDisplayName: DeLorean88
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-06-04T10:54:55.887Z
Answers.Text: You're correct: I've updated my answer.
Answers.UserDisplayName: Luca Anzalone
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-06-03T11:52:20.270Z
Answers.LastActivityDate: 2023-06-04T10:53:18.927Z
Answers.LastEditDate: 2023-06-04T10:53:18.927Z
Answers.UserDisplayName: Luca Anzalone
Body: <p>I've watched the outstanding Andrej Karpathy's <a href="https://www.youtube.com/watch?v=VMj-3S1tku0&amp;list=PLAqhIrjkxbuWI23v9cThsA9GvCAUhRvKZ&amp;index=1&amp;t=0s" rel="nofollow noreferrer">From Zero to Hero course</a>. In the last lecture, he introduces Transformer decoder architecture, which is able to produce Shakespear-like text. However, there was no direct comparison of the achieved cross-entropy loss (~1.4) with simple MLP models he talked about in the first 5 lectures.</p>
  <p>What if one trains an MLP based model with a similar number of parameters/layers, the same context length and also including layer normalization, feed forward and dropout, would the result be substantially worse? Would the training take longer? Are there direct comparisons like that in the literature?</p>
  
Tags: <transformer><multilayer-perceptrons>
Title: Are transformer models better than comparable-complexity MLP-based models?
UserDisplayName: DeLorean88
ThreadText: # Are transformer models better than comparable-complexity MLP-based models?
  
  (Asked by: username_0 on 2023-06-02)
  
  I've watched the outstanding Andrej Karpathy's [From Zero to Hero course](https://www.youtube.com/watch?v=VMj-3S1tku0&list=PLAqhIrjkxbuWI23v9cThsA9GvCAUhRvKZ&index=1&t=0s). In the last lecture, he introduces Transformer decoder architecture, which is able to produce Shakespear-like text. However, there was no direct comparison of the achieved cross-entropy loss (~1.4) with simple MLP models he talked about in the first 5 lectures.
  
  What if one trains an MLP based model with a similar number of parameters/layers, the same context length and also including layer normalization, feed forward and dropout, would the result be substantially worse? Would the training take longer? Are there direct comparisons like that in the literature?
  
  **Tags:** transformer, multilayer-perceptrons
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2023-06-03. Score: 1
  
  I can't provide you with numbers and results, but I'd expect (for not triavial problems) the MLP-based model to be worse than a Transformer.
  
  The reason is that transformers are designed to handle sequences, whereas MLP are not (even if you flatten the entire sequence into a vector). Trasformers can leverage an **inductive bias** specifically designed for sequences (likewise CNNs for images), which should account for a better learning, at least, even in case of similar number of parameters and regularization.
  
  Thanks to *self-attention* transformers are able to learn to focus on relevant parts of the sequence, even far in time, whereas simple MLPs cannot do this. Moreover, *positional encoding* provides the transformer to use the sequence order information, enabling it to learn to attend even to relative positions. Instead, MLPs just combine information from all the vectors' components.
  
  > **Comments:**
  > **username_0**: Not to nitpick, but vector components of course have ordering (it's a list, not a set, in python terminology)  
  > **username_1**: You're correct: I've updated my answer.  
  
  ---
  
  