Answers.Body: <p>BERT is deterministic. There is no variation unless you parse your tokens differently in succeeding runs. Here is the original paper the model architecture is based off of <a href="https://arxiv.org/abs/1706.03762" rel="nofollow noreferrer">Transformer Paper</a>. Note that in every layer, the only operations used for the most part are matrix multiplications, concatenations, basic ops, and layer normalizations, all of which are deterministic.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-06-04T07:25:58.357Z
Answers.Text: Thats what I thought. But when I used the method 1 mentioned in the answers for the question here : https://stackoverflow.com/questions/55369821/how-to-train-a-neural-network-model-with-bert-embeddings-instead-of-static-embed , I am getting different embeddings evrytime I run the code on same data with same tokenizer.
Answers.UserDisplayName: Srikant Jayaraman
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-06-07T22:26:19.827Z
Answers.Text: Would you share the web link of "BERT is deterministic."? Or at least where in the paper?
Answers.UserDisplayName: Cloud Cho
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-06-12T15:20:05.693Z
Answers.Text: @CloudCho "Bert is deterministic" is not a paper, but a fact derived from everything up to logits are matrix multiplications and normalizations that dont inject any randomness
Answers.UserDisplayName: mshlis
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-06-12T16:57:14.533Z
Answers.Text: That's interesting view. Would you share your definition of randomness/?  For "don't inject any randomness", is there other neural network model based on Transformer having the randomness?
Answers.UserDisplayName: Cloud Cho
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-06-12T20:37:11.360Z
Answers.Text: @CloudCho my definition of "has randomness" = can take more than one possible value with nonzero probability. and dont know off top of my head of a transformer with injected randomness, but im sure it exists. Is your goal to add priors to activations and/or weights?
Answers.UserDisplayName: mshlis
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-06-12T22:14:52.690Z
Answers.Text: Still not clear where we could make input (inject), which part of Transformer structure is/are you mentioning? If you find a Transformer based model with injected randomness, please share. Sorry for confusion, my question was to know why called deterministic.
Answers.UserDisplayName: Cloud Cho
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-06-13T11:58:24.293Z
Answers.Text: @CloudCho anywhere. e.g. y=x^2 is deterministic. but if i do y~N(x^2, .1) i added white noise to it. deterministic means no matter how many times i run it, it will produce same output. like y=x^2. if x=3, y is *always* 9
Answers.UserDisplayName: mshlis
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-06-03T15:29:43.173Z
Answers.LastActivityDate: 2019-06-03T15:29:43.173Z
Answers.UserDisplayName: mshlis
Body: <p>When we use BERT embeddings for a classification task, would we get different embeddings every time we pass the same text through the BERT architecture? If yes, is it the right way to use the embeddings as features? Ideally, while using any feature extraction technique, features values should be consistent. How do I handle this if we want BERT to be used as a feature extractor?</p>
  
Tags: <machine-learning><natural-language-processing><word-embedding><bert>
Title: Will BERT embedding be always same for a given document when used as a feature extractor
UserDisplayName: Srikant Jayaraman
ThreadText: # Will BERT embedding be always same for a given document when used as a feature extractor
  
  (Asked by: username_0 on 2019-06-03)
  
  When we use BERT embeddings for a classification task, would we get different embeddings every time we pass the same text through the BERT architecture? If yes, is it the right way to use the embeddings as features? Ideally, while using any feature extraction technique, features values should be consistent. How do I handle this if we want BERT to be used as a feature extractor?
  
  **Tags:** machine-learning, natural-language-processing, word-embedding, bert
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2019-06-03. Score: 1
  
  BERT is deterministic. There is no variation unless you parse your tokens differently in succeeding runs. Here is the original paper the model architecture is based off of [Transformer Paper](https://arxiv.org/abs/1706.03762). Note that in every layer, the only operations used for the most part are matrix multiplications, concatenations, basic ops, and layer normalizations, all of which are deterministic.
  
  > **Comments:**
  > **username_0**: Thats what I thought. But when I used the method 1 mentioned in the answers for the question here : https://stackoverflow.com/questions/55369821/how-to-train-a-neural-network-model-with-bert-embeddings-instead-of-static-embed , I am getting different embeddings evrytime I run the code on same data with same tokenizer.  
  > **username_2**: Would you share the web link of "BERT is deterministic."? Or at least where in the paper?  
  > **username_1**: @username_2 "Bert is deterministic" is not a paper, but a fact derived from everything up to logits are matrix multiplications and normalizations that dont inject any randomness  
  > **username_2**: That's interesting view. Would you share your definition of randomness/?  For "don't inject any randomness", is there other neural network model based on Transformer having the randomness?  
  > **username_1**: @username_2 my definition of "has randomness" = can take more than one possible value with nonzero probability. and dont know off top of my head of a transformer with injected randomness, but im sure it exists. Is your goal to add priors to activations and/or weights?  
  > **username_2**: Still not clear where we could make input (inject), which part of Transformer structure is/are you mentioning? If you find a Transformer based model with injected randomness, please share. Sorry for confusion, my question was to know why called deterministic.  
  > **username_1**: @username_2 anywhere. e.g. y=x^2 is deterministic. but if i do y~N(x^2, .1) i added white noise to it. deterministic means no matter how many times i run it, it will produce same output. like y=x^2. if x=3, y is *always* 9  
  
  ---
  
  