Answers.Body: <p>=> I don't have more ideas about <strong>BART</strong> and <strong>T5</strong> Right now. but I had created a chatbot Based on <strong>GPT-2</strong> Model-based on <a href="https://github.com/microsoft/DialoGPT" rel="nofollow noreferrer">Microsoft DialoGPT</a>. Which is fine-tuned on millions of parameters of Reddit. you can fine-tune on your own data using DialoGPT. I had not found a public decoding method. So, We tried to generate diverse responses using simple <strong>nucleus sampling</strong> at each time step. So, We used <strong>greedy nucleus sampling multiple times</strong> in parallel to generate multiple candidates responses. We generated 30 such responses for each turn and also used the last 3 turn conversation history as extra context for generator. So, We can generate responses related to the current Context of Conversation.</p>
  
  <p>We used Code reference from <a href="https://colab.research.google.com/drive/1-_KjlAV3J1IVDw_9KogjKDCzgFY7Jp7E#scrollTo=RZd4iXX5SqKC" rel="nofollow noreferrer">here</a>. you need to implement greedy nucleus sampling multiple on this code.</p>
  
  <p>=>Now To Choose the best-matched response from response generated I created a Reranker.I used two sub-component.</p>
  
  <p>=>Component one Counts <strong>Cosine similarity</strong> between Response and Query using <a href="https://github.com/PolyAI-LDN/polyai-models/blob/master/encoder_client.py" rel="nofollow noreferrer">sentence Encoder</a>.</p>
  
  <p>=>In Component Two we <strong>cross-entropy error</strong> for generating original query from the response.For that I used reverse-generator of <a href="https://github.com/microsoft/DialoGPT" rel="nofollow noreferrer">DialogueGPT</a>. </p>
  
  <p>We Combined Score of Both Component as <code>Score=norm(comp1Score)+norm(1-comp2Loss)</code>. Using this score you can find best response.</p>
  
  <p>For Tutorial My Favourite <a href="https://huggingface.co/transformers/" rel="nofollow noreferrer">Hugging Face</a> they have <strong>BERT,GPT, GPT-2, XLNET,etc...</strong>
  and Second Favourite <a href="https://github.com/facebookresearch/ParlAI" rel="nofollow noreferrer">Facebook ParlAI</a>, 
  try <a href="https://parl.ai/projects/recipes/" rel="nofollow noreferrer">Blender Model</a> also.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-06-12T07:14:30.290Z
Answers.Text: But can we use outputs of `BERT` as the the  `K`' and `'V` values to an attention Decoder in the simplest terms possible to minimize the `binary cross entropy`?
Answers.UserDisplayName: Deshwal
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-06-12T11:03:25.693Z
Answers.Text: @HirenNamera Can you please use `this` style only for code? You can and should use _this_ or **this** style if you need to emphasize something.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-06-12T11:10:49.960Z
Answers.Text: I think it is possible to minimize **binary cross entropy**. Why not you try using huggingface's `pretrainedencoderdecoder` module. Using **bert-base-uncased** as encoder and **gpt-2 large** as a decoder.
Answers.UserDisplayName: Hiren Namera
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-06-12T06:29:52.240Z
Answers.LastActivityDate: 2020-06-12T11:09:10.447Z
Answers.LastEditDate: 2020-06-12T11:09:10.447Z
Answers.UserDisplayName: Hiren Namera
Body: <p>I was wondering if the <code>BERT</code> or <code>T5</code> models can do the task of generating sentences in English. Most of the models I have mentioned are trained to translate from English to German or French. Is it possible that I can use the output of <code>BERT</code> as an input to my Decoder? My theory is that when I already have the trained <code>Embeddings</code>, I do not need to train the Encoder part. I can just add the outputs of sentences to the decoder to generate the sentences.</p>
  <p>In place of finding the loss value from the translated version, Can I compute loss on the reply of a sentence?</p>
  <p>Can someone point me toward a tutorial where I can use the <code>BERT</code> output for the decoder part? I have a data of conversation with me. I want to build a <code>Chatbot</code> from that data.</p>
  <p>I have already implemented <code>LSTM</code> based <code>Sequence2sequence</code> model but it is not providing me satisfactory answer.</p>
  <p>After some research, 2 such models are there as <code>T5</code> and <code>BART</code> which are based on the same idea.</p>
  <p>If possible, can someone tell me how can I use <code>BART</code> or <code>T5</code> to make a conversational bot?</p>
  
Tags: <machine-learning><deep-learning><natural-language-processing><deep-neural-networks><bert>
Title: Can we use a pre trained Encoder (BERT, XLM ) with a Decoder (GPT, Transformer-XL) to build a Chatbot instead of Language Translation?
UserDisplayName: Deshwal
ThreadText: # Can we use a pre trained Encoder (BERT, XLM ) with a Decoder (GPT, Transformer-XL) to build a Chatbot instead of Language Translation?
  
  (Asked by: username_0 on 2020-06-12)
  
  I was wondering if the `BERT` or `T5` models can do the task of generating sentences in English. Most of the models I have mentioned are trained to translate from English to German or French. Is it possible that I can use the output of `BERT` as an input to my Decoder? My theory is that when I already have the trained `Embeddings`, I do not need to train the Encoder part. I can just add the outputs of sentences to the decoder to generate the sentences.
  
  In place of finding the loss value from the translated version, Can I compute loss on the reply of a sentence?
  
  Can someone point me toward a tutorial where I can use the `BERT` output for the decoder part? I have a data of conversation with me. I want to build a `Chatbot` from that data.
  
  I have already implemented `LSTM` based `Sequence2sequence` model but it is not providing me satisfactory answer.
  
  After some research, 2 such models are there as `T5` and `BART` which are based on the same idea.
  
  If possible, can someone tell me how can I use `BART` or `T5` to make a conversational bot?
  
  **Tags:** machine-learning, deep-learning, natural-language-processing, deep-neural-networks, bert
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2020-06-12. Score: 1
  
  => I don't have more ideas about **BART** and **T5** Right now. but I had created a chatbot Based on **GPT-2** Model-based on [Microsoft DialoGPT](https://github.com/microsoft/DialoGPT). Which is fine-tuned on millions of parameters of Reddit. you can fine-tune on your own data using DialoGPT. I had not found a public decoding method. So, We tried to generate diverse responses using simple **nucleus sampling** at each time step. So, We used **greedy nucleus sampling multiple times** in parallel to generate multiple candidates responses. We generated 30 such responses for each turn and also used the last 3 turn conversation history as extra context for generator. So, We can generate responses related to the current Context of Conversation.
  
  We used Code reference from [here](https://colab.research.google.com/drive/1-_KjlAV3J1IVDw_9KogjKDCzgFY7Jp7E#scrollTo=RZd4iXX5SqKC). you need to implement greedy nucleus sampling multiple on this code.
  
  =>Now To Choose the best-matched response from response generated I created a Reranker.I used two sub-component.
  
  =>Component one Counts **Cosine similarity** between Response and Query using [sentence Encoder](https://github.com/PolyAI-LDN/polyai-models/blob/master/encoder_client.py).
  
  =>In Component Two we **cross-entropy error** for generating original query from the response.For that I used reverse-generator of [DialogueGPT](https://github.com/microsoft/DialoGPT).
  
  We Combined Score of Both Component as `Score=norm(comp1Score)+norm(1-comp2Loss)`. Using this score you can find best response.
  
  For Tutorial My Favourite [Hugging Face](https://huggingface.co/transformers/) they have **BERT,GPT, GPT-2, XLNET,etc...**
  and Second Favourite [Facebook ParlAI](https://github.com/facebookresearch/ParlAI),
  try [Blender Model](https://parl.ai/projects/recipes/) also.
  
  > **Comments:**
  > **username_0**: But can we use outputs of `BERT` as the the  `K`' and `'V` values to an attention Decoder in the simplest terms possible to minimize the `binary cross entropy`?  
  > **username_2**: @HirenNamera Can you please use `this` style only for code? You can and should use _this_ or **this** style if you need to emphasize something.  
  > **username_1**: I think it is possible to minimize **binary cross entropy**. Why not you try using huggingface's `pretrainedencoderdecoder` module. Using **bert-base-uncased** as encoder and **gpt-2 large** as a decoder.  
  
  ---
  
  