Answers.Body: <p>You can follow this example code <a href="https://github.com/microsoft/CodeBERT/blob/master/CodeBERT/code2nl/README.md" rel="nofollow noreferrer">https://github.com/microsoft/CodeBERT/blob/master/CodeBERT/code2nl/README.md</a></p>
  <p>This example generally loads pre-trained Bert (encoder) and plugs a custom decoder. Regarding its encoder, the example utilizes <code>transformer</code> for loading pre-train.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-07-21T15:46:05.013Z
Answers.LastActivityDate: 2023-07-21T15:46:05.013Z
Answers.UserDisplayName: Bryan
Body: <p><strong>Context:</strong>
  I am currently working on an encoder-decoder sequence to sequence model that uses a sequence of word embeddings as input and output, and then reduces the dimensionality of the word embeddings.</p>
  <p>The word embeddings are created using pre-trained models. I want to be able to decode the word embeddings returned by the decoder of the Sequence to Sequence model back to natural language.</p>
  <p><strong>Question:</strong> How can I train a Decoder that works with the sequence of word embeddings and the original sentence for this task?</p>
  <p>See below for the code that generates the word embeddings:</p>
  <pre><code>from typing import List
  
  import numpy as np
  import torch
  from transformers.tokenization_utils_base import BatchEncoding
  from transformers import BertTokenizerFast, BertModel
  
  TOKENIZER = BertTokenizerFast.from_pretrained('bert-base-uncased')
  MODEL = BertModel.from_pretrained('bert-base-uncased')
  
  def get_word_indices(sentence: str, separator=&quot; &quot;) -&gt; List:
      sent = sentence.split(sep=separator)
      return list(range(len(sent)))
  
  def encode_sentence(sentence: str) -&gt; BatchEncoding:
      encoded_sentence = TOKENIZER(sentence)
      return encoded_sentence
  
  def get_hidden_states(encoded: BatchEncoding, layers: list = [-1, -2, -3, -4]) -&gt; torch.Tensor:
      with torch.no_grad():
          output = MODEL(**encoded)
      hidden_states = output.hidden_states
      output = torch.stack([hidden_states[i] for i in layers]).sum(0).squeeze()
      return output
  
  def get_token_ids(word_index: int, encoded: BatchEncoding):
      token_ids = np.where(np.array(encoded.word_ids()) == word_index)
      return token_ids
  
  def embed_words(sentence: str) -&gt; torch.Tensor:
      word_indices = get_word_indices(sentence)
      encoded_sentence = encode_sentence(sentence)
      hidden_states = get_hidden_states(encoded_sentence)
      word_embeddings = []
      for word_index in word_indices:
          # Get the ids of the word in the sentence
          # Important, because BERT sometimes splits words into subwords
          token_ids = get_token_ids(word_index, encoded_sentence)
          # Get all the hidden states for each word (or subwords belonging to one word)
          # Average the hidden states in case of subwords to retrieve word embedding
          word_embedding = hidden_states[token_ids].mean(dim=0)
          word_embeddings.append(word_embedding)
      return torch.stack(word_embeddings)
  <span class="math-container">```</span>
  </code></pre>
  
Tags: <transformer><bert><pretrained-models><seq2seq><encoder-decoder>
Title: How to Train a Decoder for Pre-trained BERT Transformer-Encoder?
UserDisplayName: node_env
ThreadText: # How to Train a Decoder for Pre-trained BERT Transformer-Encoder?
  
  (Asked by: username_0 on 2022-09-01)
  
  Context:
  I am currently working on an encoder-decoder sequence to sequence model that uses a sequence of word embeddings as input and output, and then reduces the dimensionality of the word embeddings.
  The word embeddings are created using pre-trained models. I want to be able to decode the word embeddings returned by the decoder of the Sequence to Sequence model back to natural language.
  Question: How can I train a Decoder that works with the sequence of word embeddings and the original sentence for this task?
  See below for the code that generates the word embeddings:
  from typing import List
  
  import numpy as np
  import torch
  from transformers.tokenization_utils_base import BatchEncoding
  from transformers import BertTokenizerFast, BertModel
  
  TOKENIZER = BertTokenizerFast.from_pretrained('bert-base-uncased')
  MODEL = BertModel.from_pretrained('bert-base-uncased')
  
  def get_word_indices(sentence: str, separator=" ") -> List:
      sent = sentence.split(sep=separator)
      return list(range(len(sent)))
  
  def encode_sentence(sentence: str) -> BatchEncoding:
      encoded_sentence = TOKENIZER(sentence)
      return encoded_sentence
  
  def get_hidden_states(encoded: BatchEncoding, layers: list = [-1, -2, -3, -4]) -> torch.Tensor:
      with torch.no_grad():
          output = MODEL(**encoded)
      hidden_states = output.hidden_states
      output = torch.stack([hidden_states[i] for i in layers]).sum(0).squeeze()
      return output
  
  def get_token_ids(word_index: int, encoded: BatchEncoding):
      token_ids = np.where(np.array(encoded.word_ids()) == word_index)
      return token_ids
  
  def embed_words(sentence: str) -> torch.Tensor:
      word_indices = get_word_indices(sentence)
      encoded_sentence = encode_sentence(sentence)
      hidden_states = get_hidden_states(encoded_sentence)
      word_embeddings = []
      for word_index in word_indices:
          # Get the ids of the word in the sentence
          # Important, because BERT sometimes splits words into subwords
          token_ids = get_token_ids(word_index, encoded_sentence)
          # Get all the hidden states for each word (or subwords belonging to one word)
          # Average the hidden states in case of subwords to retrieve word embedding
          word_embedding = hidden_states[token_ids].mean(dim=0)
          word_embeddings.append(word_embedding)
      return torch.stack(word_embeddings)
  ```
  
  **Tags:** transformer, bert, pretrained-models, seq2seq, encoder-decoder
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2023-07-21. Score: 0
  
  You can follow this example code <https://github.com/microsoft/CodeBERT/blob/master/CodeBERT/code2nl/README.md>
  
  This example generally loads pre-trained Bert (encoder) and plugs a custom decoder. Regarding its encoder, the example utilizes `transformer` for loading pre-train.
  
  
  ---
  
  