Answers.Body: <p>This seems like a problem for the use of an encoder-decoder pair such as those seen in text summarization (see this paper by Rush et al.: <a href="https://arxiv.org/pdf/1509.00685.pdf%C3%AF%C2%BC" rel="nofollow noreferrer">https://arxiv.org/pdf/1509.00685.pdfï%C2%BC</a>). </p>
  
  <p>You would need the following layers: </p>
  
  <ul>
  <li><p>LSTM layer to encode the given input text into an embedding </p></li>
  <li><p>LSTM layer the looks over the currently generated output to encode the text into an embedding</p></li>
  <li><p>A dense soft-max layer for generating words probabilistically based on the output of the two contextual LSTM encoders</p></li>
  </ul>
  
  <p>Please see the following blog post by Jason Brownlee that outlines this approach and others, while giving implementation details and snippets (<a href="https://machinelearningmastery.com/encoder-decoder-models-text-summarization-keras/" rel="nofollow noreferrer">https://machinelearningmastery.com/encoder-decoder-models-text-summarization-keras/</a>)! </p>
  
  <p>Also note that this would require a large set of training examples of input text and reasonable responses. You might be able to scrape Reddit post responses and comments off of those for a start? Let me know if I misunderstood the question.</p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2018-04-06T11:49:26.180Z
Answers.Text: You did not misunderstand the question, but I am using JS, and Synaptic.js at that. The documentation is [here](https://github.com/cazala/synaptic/wiki). I am unable to use Keras, for I am not using Python.
Answers.UserDisplayName: FreezePhoenix
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2018-04-06T12:03:13.217Z
Answers.Text: Wait.... That image looks familiar: https://3qeqpr26caki16dnhd19sv6by6v-wpengine.netdna-ssl.com/wp-content/uploads/2017/09/General-Text-Summarization-Model-in-Keras.png
Answers.UserDisplayName: FreezePhoenix
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2018-04-06T12:05:04.097Z
Answers.Text: I think I have it down. Just one more question: Does the embedding layer mean that it turns text into something the NN can understand, such as floating point numbers, while the output layer turns that back into text?
Answers.UserDisplayName: FreezePhoenix
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2018-04-06T15:52:16.187Z
Answers.Text: @Pheo, yes you are correct, it turns each word into a stream of numbers, often times a vector. This is what Word2Vec and GloVe does, which you can load into your model in python. And then the last layer looks at these and can choose the output in a variety of ways, these are known as decoders. I apologize I'm not very familiar with JS, but if you have any conceptual questions I should be able to help out.
Answers.UserDisplayName: JMed
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2018-04-06T17:47:45.130Z
Answers.Text: Conceptual question: Is the vector an array of indexes or an array representation of the Unicode values?
Answers.UserDisplayName: FreezePhoenix
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2018-04-07T19:42:55.813Z
Answers.Text: Neither, it actually is a vector representing the symantic meaning of a word. That's why many people choose to make their own embedding through training or use Google's Word2Vec or GloVe since they've already been trained over a large set
Answers.UserDisplayName: JMed
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2018-04-06T00:38:01.823Z
Answers.LastActivityDate: 2018-04-06T00:38:01.823Z
Answers.UserDisplayName: JMed
Body: <p>I have recently gone about and made a <a href="https://jsbin.com/vijawuleyo/edit?html,console,output" rel="nofollow noreferrer">simple AI</a>, one that gives responses to an input (albeit completely irrelevant and nonsensical ones), using Synaptic.js. Unfortunately, this is not the type of text generation I am looking for. What I am looking for would be a way to get connections between words and generate text from that. (What would be preferable would be to also generate at least semi-sensible answers also.)</p>
  
  <p>This is part of project Raphiel, and can be checked out in the room associated with this site. What I want to know is what layer combination would I use for text generation?</p>
  
  <p>I have been told to avoid retrieval-based bots.</p>
  
  <p>I have the method to send and receive messages, I just need to figure out what combination of layers would be the best. </p>
  
  <p>Unless I have the numbers wrong, this will be SE's second NN chatbot.</p>
  
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2018-04-05T13:35:07.640Z
Comments.Text: To run the simple AI, please type a vocab into the text area, click `load`, and run in the console `init()`. Then, you can click the "test" button.
Comments.UserDisplayName: FreezePhoenix
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2018-04-05T13:42:21.127Z
Comments.Text: Oh, and Raphiel is currently running under my account.
Comments.UserDisplayName: FreezePhoenix
Tags: <ai-design><text-summarization>
Title: How would one go about generating *sensible* responses to chat?
UserDisplayName: FreezePhoenix
ThreadText: # How would one go about generating *sensible* responses to chat?
  
  (Asked by: username_0 on 2018-04-05)
  
  I have recently gone about and made a [simple AI](https://jsbin.com/vijawuleyo/edit?html,console,output), one that gives responses to an input (albeit completely irrelevant and nonsensical ones), using Synaptic.js. Unfortunately, this is not the type of text generation I am looking for. What I am looking for would be a way to get connections between words and generate text from that. (What would be preferable would be to also generate at least semi-sensible answers also.)
  
  This is part of project Raphiel, and can be checked out in the room associated with this site. What I want to know is what layer combination would I use for text generation?
  
  I have been told to avoid retrieval-based bots.
  
  I have the method to send and receive messages, I just need to figure out what combination of layers would be the best.
  
  Unless I have the numbers wrong, this will be SE's second NN chatbot.
  
  **Tags:** ai-design, text-summarization
  
  > **Comments:**
  > **username_0**: To run the simple AI, please type a vocab into the text area, click `load`, and run in the console `init()`. Then, you can click the "test" button.  
  > **username_0**: Oh, and Raphiel is currently running under my account.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2018-04-06. Score: 2 | Accepted answer
  
  This seems like a problem for the use of an encoder-decoder pair such as those seen in text summarization (see this paper by Rush et al.: [https://arxiv.org/pdf/1509.00685.pdfï%C2%BC](https://arxiv.org/pdf/1509.00685.pdf%C3%AF%C2%BC)).
  
  You would need the following layers:
  
  * LSTM layer to encode the given input text into an embedding
  * LSTM layer the looks over the currently generated output to encode the text into an embedding
  * A dense soft-max layer for generating words probabilistically based on the output of the two contextual LSTM encoders
  
  Please see the following blog post by Jason Brownlee that outlines this approach and others, while giving implementation details and snippets (<https://machinelearningmastery.com/encoder-decoder-models-text-summarization-keras/>)!
  
  Also note that this would require a large set of training examples of input text and reasonable responses. You might be able to scrape Reddit post responses and comments off of those for a start? Let me know if I misunderstood the question.
  
  > **Comments:**
  > **username_0**: You did not misunderstand the question, but I am using JS, and Synaptic.js at that. The documentation is [here](https://github.com/cazala/synaptic/wiki). I am unable to use Keras, for I am not using Python.  
  > **username_0**: Wait.... That image looks familiar: https://3qeqpr26caki16dnhd19sv6by6v-wpengine.netdna-ssl.com/wp-content/uploads/2017/09/General-Text-Summarization-Model-in-Keras.png  
  > **username_0**: I think I have it down. Just one more question: Does the embedding layer mean that it turns text into something the NN can understand, such as floating point numbers, while the output layer turns that back into text?  
  > **username_1**: @Pheo, yes you are correct, it turns each word into a stream of numbers, often times a vector. This is what Word2Vec and GloVe does, which you can load into your model in python. And then the last layer looks at these and can choose the output in a variety of ways, these are known as decoders. I apologize I'm not very familiar with JS, but if you have any conceptual questions I should be able to help out.  
  > **username_0**: Conceptual question: Is the vector an array of indexes or an array representation of the Unicode values?  
  > **username_1**: Neither, it actually is a vector representing the symantic meaning of a word. That's why many people choose to make their own embedding through training or use Google's Word2Vec or GloVe since they've already been trained over a large set  
  
  ---
  
  