Answers.Body: <p>I'd expect your approach to converge faster indeed.</p>
  <p>Not having to start from scratch regarding your embedding vectors means that there are fewer parameters to learn. Assuming that the token embeddings are indeed useful for the model, the model should converge faster. However, note that a trainable token embedding might eventually exceed the model with a pre-trained token embedding with regard to performance.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-07-08T18:01:45.083Z
Answers.Text: What about a fine-tuning approach, whereby the original pre-trained token/word embeddings can be finetuned to the task
Answers.UserDisplayName: information_interchange
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-07-09T08:54:48.683Z
Answers.Text: You migjt then wany to start with frozen embeddings, and slowly unfreeze them. Otherwise you might get some catastrophic forgetting
Answers.UserDisplayName: Robin van Hoorn
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-05-15T11:35:04.627Z
Answers.LastActivityDate: 2023-05-15T11:35:04.627Z
Answers.UserDisplayName: Robin van Hoorn
Body: <p>I read the answers for that question <a href="https://ai.stackexchange.com/questions/26235/what-kind-of-word-embedding-is-used-in-the-original-transformer">What kind of word embedding is used in the original transformer?</a>. It says that transforms like bert start the first word embedding layer with random values.</p>
  <p>Initializing the first word embedding layer in transformers with random values works fine but Wouldn't initializing transformers with pre-trained word embedding speed up the training of transformers?</p>
  <p>Isn't starting with pre-trained word embedding(vectors that have semantic meaning) is better than starting from scratch?</p>
  <p>I am not talking about the performance. I am talking about the speed of the training.</p>
  
Tags: <deep-learning><transformer><word-embedding>
Title: Would initializing transformers with pre-trained word embedding speed up the training of transformers?
UserDisplayName: floyd
ThreadText: # Would initializing transformers with pre-trained word embedding speed up the training of transformers?
  
  (Asked by: username_0 on 2023-05-13)
  
  I read the answers for that question [What kind of word embedding is used in the original transformer?](https://ai.stackexchange.com/questions/26235/what-kind-of-word-embedding-is-used-in-the-original-transformer). It says that transforms like bert start the first word embedding layer with random values.
  
  Initializing the first word embedding layer in transformers with random values works fine but Wouldn't initializing transformers with pre-trained word embedding speed up the training of transformers?
  
  Isn't starting with pre-trained word embedding(vectors that have semantic meaning) is better than starting from scratch?
  
  I am not talking about the performance. I am talking about the speed of the training.
  
  **Tags:** deep-learning, transformer, word-embedding
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2023-05-15. Score: 2
  
  I'd expect your approach to converge faster indeed.
  
  Not having to start from scratch regarding your embedding vectors means that there are fewer parameters to learn. Assuming that the token embeddings are indeed useful for the model, the model should converge faster. However, note that a trainable token embedding might eventually exceed the model with a pre-trained token embedding with regard to performance.
  
  > **Comments:**
  > **username_2**: What about a fine-tuning approach, whereby the original pre-trained token/word embeddings can be finetuned to the task  
  > **username_1**: You migjt then wany to start with frozen embeddings, and slowly unfreeze them. Otherwise you might get some catastrophic forgetting  
  
  ---
  
  