Answers.Body: <p>A neural network (NN) is a "function approximator", that is, it is a model that can be used to approximate functions. In fact, a neural network with at least one hidden layer is a <a href="http://neuralnetworksanddeeplearning.com/chap4.html" rel="nofollow noreferrer">"universal" function approximator</a> (that is, it can approximate any function).</p>
  
  <p>In mathematics, a function <span class="math-container">$f$</span> is usually represented as a mapping of the form <span class="math-container">$f: D \rightarrow C$</span>, where <span class="math-container">$D$</span> and <span class="math-container">$C$</span> are respectively the <a href="https://en.wikipedia.org/wiki/Domain_of_a_function" rel="nofollow noreferrer">domain</a> (inputs) and <a href="https://en.wikipedia.org/wiki/Codomain" rel="nofollow noreferrer">codomain</a> (outputs) of <span class="math-container">$f$</span>, and <span class="math-container">$\rightarrow$</span> means that <span class="math-container">$f$</span> "maps" <span class="math-container">$D$</span> to <span class="math-container">$C$</span>. A NN (with at least one hidden layer) can thus approximate any function <span class="math-container">$f$</span> of this form.</p>
  
  <p>In your context, the <span class="math-container">$Q$</span> table is a function: it is a mapping between states and actions (inputs) and Q values (outputs), which are the "expected future cumulative reward" (that you will obtain if you take a certain action from a certain state and then continue to follow the same policy). The Q function can thus more formally be denoted by <span class="math-container">$Q: (S, A) \rightarrow \mathcal{R}$</span>, where <span class="math-container">$S$</span> is the "space of states" and <span class="math-container">$A$</span> is the "space of actions" in your problem. Initially, your Q table does not contain the correct (optimal) values. However, after training (or learning), hopefully, your Q table will be an approximation to the optimal Q function for your specific problem.</p>
  
  <p>How do you then represent this table using a NN? What should be the inputs and outputs of the NN?</p>
  
  <p>Let's suppose that your <span class="math-container">$Q$</span> table is implemented as matrix <span class="math-container">$M$</span>. Then <span class="math-container">$M[s, a]$</span> is the <span class="math-container">$Q$</span> value for the state <span class="math-container">$s$</span> and action <span class="math-container">$a$</span>. So, in this case, the combination of <span class="math-container">$s$</span> and <span class="math-container">$a$</span> is the input, whereas <span class="math-container">$M[s, a]$</span> is the output of your <span class="math-container">$Q$</span> function.  </p>
  
  <p>To represent this table as a NN, you can thus have as input of the NN the state and action, and as output the <span class="math-container">$Q$</span> value. You will then train the NN (using e.g. back-propagation) to learn the <span class="math-container">$Q(s, a)$</span>, given a state <span class="math-container">$s$</span> and an action <span class="math-container">$a$</span> as input, for all <span class="math-container">$s \in \mathcal{S}$</span> and <span class="math-container">$a \in \mathcal{A}$</span>. So, during your <span class="math-container">$Q$</span>-learning algorithm, instead of using <span class="math-container">$M[s, a]$</span> to represent <span class="math-container">$Q(s, a)$</span>, you will simply use the current output of your neural network. </p>
  
  <p>Note that, in practice, I think, you will likely encounter problems while training the NN, if you update the weights of your NN at every time step (because e.g. NNs do not cope well with correlated data, and, in general, the "experience" data you will obtain from time step to time step will be highly correlated). Anyway, this is the basic idea of how to use a NN to represent a <span class="math-container">$Q$</span> function. There are other ways, but this is the simplest one, at least, conceptually.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-02-09T16:37:52.033Z
Answers.Text: I've used 2D-array that represent Q-Matrix and all elements of array are iterating to form a string that's called state of Q-Matrix.So how can I set state as input of neuron while neuron input must be in float type and weight values are also used between -1 and 1 that is to update later on.
Answers.UserDisplayName: Arslan Ali
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-02-09T21:29:28.273Z
Answers.Text: Why are you creating a string out of the 2d array?
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-02-10T08:20:06.947Z
Answers.Text: So,How can I create states?It's represented matrix state where the agent exist recently and I'm adding it into list with resizing of matrix in case of dynamic states or newer states.
Answers.UserDisplayName: Arslan Ali
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-02-10T08:22:03.283Z
Answers.Text: State of matrix can not be presented in floating points.It's possible only for distance based applications or other related to positions of agents in 2d-3d surfaces.
Answers.UserDisplayName: Arslan Ali
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-02-10T09:12:08.453Z
Answers.Text: how to get state of matrix in float Number
Answers.UserDisplayName: Arslan Ali
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-02-10T11:41:58.613Z
Answers.Text: @ArslanAli I am not sure I understand what you just said above. Why can't you create a matrix of floating point numbers?
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-02-10T12:03:26.383Z
Answers.Text: I'm saying how I will send state as input while state and action must be float number with respect to weights.
Answers.UserDisplayName: Arslan Ali
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-02-10T12:08:21.573Z
Answers.Text: @ArslanAli You could e.g. encode your actions (up, down, etc.) as numbers: e.g. up could be 0, down could be 1, and so on. The states could e.g. be encoded as the indices you use to access the state. You could then combine actions and states (which, at this point, should be numbers) in a vector. I am not sure if this is the best way to do it, but this could be a way to do it.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-02-10T12:12:34.850Z
Answers.Text: But input for NN must be between 0 and 1 while states can be greater than 1 in record while four action are choiced 0-3.So how these inputs can be given to NN while weights are also between 0 and 1 with respect to each input.
Answers.UserDisplayName: Arslan Ali
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-02-10T12:29:56.480Z
Answers.Text: Rows are states while columns are actions in Q-Table.So,in that case where I have to save the states and corresponding learning Q-Values in NN.
Answers.UserDisplayName: Arslan Ali
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-02-10T13:16:48.657Z
Answers.Text: @ArslanAli Each input to a NN must be a real number number (a scalar). But you can have more than one input to the NN. In that case, you can pack these numbers into a vector.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-02-10T13:20:12.323Z
Answers.Text: Output will not be checked while updating weights
Answers.UserDisplayName: Arslan Ali
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-02-10T13:49:58.163Z
Answers.Text: @ArslanAli What do you mean? What does this have to do with what I said?
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-02-10T13:55:06.567Z
Answers.Text: I have to pack each action and state in vector then send as input?
Answers.UserDisplayName: Arslan Ali
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-02-10T13:56:12.603Z
Answers.Text: As new states will come then I have to increase input in NN.That's meant by you
Answers.UserDisplayName: Arslan Ali
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-02-10T14:04:32.500Z
Answers.Text: @ArslanAli Yes, I suggested that you could pack the state and the action (both encoded as real numbers) in a vector, which would be the input to the NN. As I explain in my answer above, this NN would represent your $Q(s, a)$ function. So, at every point of your Q-learning algorithm, the $Q$ function only requires two inputs ($s$ and $a$), so I am not sure why you're saying that the inputs to the NN would increase. The dimensions of the inputs remain fixed. What changes is the actual input (i.e. the encoded $s$ and $a$) that you pass.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-02-10T14:16:50.287Z
Answers.Text: It will require new table while pre results will be saved somewhere.It is because when I will find Max from Q(s',a') that will be needed for equation.
Answers.UserDisplayName: Arslan Ali
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-02-10T14:24:48.510Z
Answers.Text: @ArslanAli No, you will not require a new table. You may want to use two NNs (where one for the target). Please, read the paper "[Human-level control through deep reinforcement learning](https://web.stanford.edu/class/psych209/Readings/MnihEtAlHassibis15NatureControlDeepRL.pdf)" by DeepMind. There you will find an explanation of a network which is used to represent the Q function.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-02-08T17:48:56.533Z
Answers.LastActivityDate: 2019-02-08T21:10:38.877Z
Answers.LastEditDate: 2019-02-08T21:10:38.877Z
Answers.UserDisplayName: nbro
Body: <p>I've used a table to represent the Q function, while an agent is being trained to catch the cheese without touching the walls.</p>
  
  <p>The first and last row (and column) of the matrix are associated with the walls. I placed in last cell a cheese that agent must catch while being training.</p>
  
  <p>So far, I've done it with dynamic states and, when necessary, I resized matrix with new states. I've used four actions (up, left, right and down). </p>
  
  <p>I would like now to use an ANN to represent my Q function. How do I do that? What should be the input and output of such neural network?</p>
  
Tags: <neural-networks><reinforcement-learning><q-learning>
Title: How do I convert table-based to neural network-based Q-learning?
UserDisplayName: Arslan Ali
ThreadText: # How do I convert table-based to neural network-based Q-learning?
  
  (Asked by: username_0 on 2019-02-08)
  
  I've used a table to represent the Q function, while an agent is being trained to catch the cheese without touching the walls.
  
  The first and last row (and column) of the matrix are associated with the walls. I placed in last cell a cheese that agent must catch while being training.
  
  So far, I've done it with dynamic states and, when necessary, I resized matrix with new states. I've used four actions (up, left, right and down).
  
  I would like now to use an ANN to represent my Q function. How do I do that? What should be the input and output of such neural network?
  
  **Tags:** neural-networks, reinforcement-learning, q-learning
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2019-02-08. Score: 1
  
  A neural network (NN) is a "function approximator", that is, it is a model that can be used to approximate functions. In fact, a neural network with at least one hidden layer is a ["universal" function approximator](http://neuralnetworksanddeeplearning.com/chap4.html) (that is, it can approximate any function).
  
  In mathematics, a function $f$ is usually represented as a mapping of the form $f: D \rightarrow C$, where $D$ and $C$ are respectively the [domain](https://en.wikipedia.org/wiki/Domain_of_a_function) (inputs) and [codomain](https://en.wikipedia.org/wiki/Codomain) (outputs) of $f$, and $\rightarrow$ means that $f$ "maps" $D$ to $C$. A NN (with at least one hidden layer) can thus approximate any function $f$ of this form.
  
  In your context, the $Q$ table is a function: it is a mapping between states and actions (inputs) and Q values (outputs), which are the "expected future cumulative reward" (that you will obtain if you take a certain action from a certain state and then continue to follow the same policy). The Q function can thus more formally be denoted by $Q: (S, A) \rightarrow \mathcal{R}$, where $S$ is the "space of states" and $A$ is the "space of actions" in your problem. Initially, your Q table does not contain the correct (optimal) values. However, after training (or learning), hopefully, your Q table will be an approximation to the optimal Q function for your specific problem.
  
  How do you then represent this table using a NN? What should be the inputs and outputs of the NN?
  
  Let's suppose that your $Q$ table is implemented as matrix $M$. Then $M[s, a]$ is the $Q$ value for the state $s$ and action $a$. So, in this case, the combination of $s$ and $a$ is the input, whereas $M[s, a]$ is the output of your $Q$ function.
  
  To represent this table as a NN, you can thus have as input of the NN the state and action, and as output the $Q$ value. You will then train the NN (using e.g. back-propagation) to learn the $Q(s, a)$, given a state $s$ and an action $a$ as input, for all $s \in \mathcal{S}$ and $a \in \mathcal{A}$. So, during your $Q$-learning algorithm, instead of using $M[s, a]$ to represent $Q(s, a)$, you will simply use the current output of your neural network.
  
  Note that, in practice, I think, you will likely encounter problems while training the NN, if you update the weights of your NN at every time step (because e.g. NNs do not cope well with correlated data, and, in general, the "experience" data you will obtain from time step to time step will be highly correlated). Anyway, this is the basic idea of how to use a NN to represent a $Q$ function. There are other ways, but this is the simplest one, at least, conceptually.
  
  > **Comments:**
  > **username_0**: I've used 2D-array that represent Q-Matrix and all elements of array are iterating to form a string that's called state of Q-Matrix.So how can I set state as input of neuron while neuron input must be in float type and weight values are also used between -1 and 1 that is to update later on.  
  > **username_1**: Why are you creating a string out of the 2d array?  
  > **username_0**: So,How can I create states?It's represented matrix state where the agent exist recently and I'm adding it into list with resizing of matrix in case of dynamic states or newer states.  
  > **username_0**: State of matrix can not be presented in floating points.It's possible only for distance based applications or other related to positions of agents in 2d-3d surfaces.  
  > **username_0**: how to get state of matrix in float Number  
  > **username_1**: @username_0 I am not sure I understand what you just said above. Why can't you create a matrix of floating point numbers?  
  > **username_0**: I'm saying how I will send state as input while state and action must be float number with respect to weights.  
  > **username_1**: @username_0 You could e.g. encode your actions (up, down, etc.) as numbers: e.g. up could be 0, down could be 1, and so on. The states could e.g. be encoded as the indices you use to access the state. You could then combine actions and states (which, at this point, should be numbers) in a vector. I am not sure if this is the best way to do it, but this could be a way to do it.  
  > **username_0**: But input for NN must be between 0 and 1 while states can be greater than 1 in record while four action are choiced 0-3.So how these inputs can be given to NN while weights are also between 0 and 1 with respect to each input.  
  > **username_0**: Rows are states while columns are actions in Q-Table.So,in that case where I have to save the states and corresponding learning Q-Values in NN.  
  > **username_1**: @username_0 Each input to a NN must be a real number number (a scalar). But you can have more than one input to the NN. In that case, you can pack these numbers into a vector.  
  > **username_0**: Output will not be checked while updating weights  
  > **username_1**: @username_0 What do you mean? What does this have to do with what I said?  
  > **username_0**: I have to pack each action and state in vector then send as input?  
  > **username_0**: As new states will come then I have to increase input in NN.That's meant by you  
  > **username_1**: @username_0 Yes, I suggested that you could pack the state and the action (both encoded as real numbers) in a vector, which would be the input to the NN. As I explain in my answer above, this NN would represent your $Q(s, a)$ function. So, at every point of your Q-learning algorithm, the $Q$ function only requires two inputs ($s$ and $a$), so I am not sure why you're saying that the inputs to the NN would increase. The dimensions of the inputs remain fixed. What changes is the actual input (i.e. the encoded $s$ and $a$) that you pass.  
  > **username_0**: It will require new table while pre results will be saved somewhere.It is because when I will find Max from Q(s',a') that will be needed for equation.  
  > **username_1**: @username_0 No, you will not require a new table. You may want to use two NNs (where one for the target). Please, read the paper "[Human-level control through deep reinforcement learning](https://web.stanford.edu/class/psych209/Readings/MnihEtAlHassibis15NatureControlDeepRL.pdf)" by DeepMind. There you will find an explanation of a network which is used to represent the Q function.  
  
  ---
  
  