Answers.Body: <p>Neural networks normally work in continuous spaces. A typical neural network function could be written as <span class="math-container">$f(\mathbf{x}, \mathbf{\theta}): \mathbb{R}^N \rightarrow \mathbb{R}^M$</span>. That is, a function of some <span class="math-container">$N$</span> dimensional input vector of real numbers <span class="math-container">$\mathbf{x}$</span> that outputs some <span class="math-container">$M$</span> dimensional output vector of real numbers (which you could call <span class="math-container">$\mathbf{y}$</span>) and that is parametrised by a vector of weights and biases <span class="math-container">$\mathbf{\theta}$</span>.</p>
  <p>When you train a neural network with some data <span class="math-container">$\mathbf{X}, \mathbf{Y}$</span> then you are trying to adjust <span class="math-container">$\mathbf{\theta}$</span> so that <span class="math-container">$f(\mathbf{x}, \mathbf{\theta})$</span> approximates some assumed mapping function <span class="math-container">$g(\mathbf{x}): \mathcal{X} \rightarrow \mathcal{Y}$</span> that provides a &quot;true&quot; mapping between all possible inputs <span class="math-container">$\mathcal{X}$</span> and the matching correct values from all possible outputs <span class="math-container">$\mathcal{Y}$</span>. In a neural network, then <span class="math-container">$\mathcal{X}$</span> and <span class="math-container">$\mathcal{Y}$</span> must be sets of vectors of real numbers.</p>
  <p>So, yes all neural networks map between continuous values by default*. When NNs are used to work with discrete values, then the discrete variables need to be converted to real-valued variables in order to be usable with a NN. So discrete inputs might be mapped to a small set of different real values or a vector of <span class="math-container">$\{0, 1 \}$</span> (called &quot;one hot&quot; encoding). Discrete outputs are often mapped to a probability vector giving the confidence that the learned function assigns to each possible discrete output - this is the usual case in classifiers.</p>
  <p>You have a specific use case in mind:</p>
  <blockquote>
  <p>More specifically, I would like to use neural networks for learning the Q-function in Reinforcement Learning. My problem has basically a continuous state and action space, and I would like to know whether I have to discretize it or not</p>
  </blockquote>
  <p>For an action value, or Q function in reinforcement learning, then you do not need to be concerned that state space or action space are continuous. However, there are some details that may be important:</p>
  <ul>
  <li><p>Neural networks learn most efficiently when input elements are within a unit-like distribution such as <span class="math-container">$\mathcal{N}(0,1)$</span> (normal distribution with mean <span class="math-container">$0$</span>, standard deviation <span class="math-container">$1$</span>). This does not have to be precise, but you should take care to scale the input values so that each element has a typical magnitude around <span class="math-container">$1$</span>.</p>
  </li>
  <li><p>The output values you want to learn need to be within the domain of the output layer's activation function. It is common to use linear (or &quot;no&quot; activation function) in the output layer for regression problems for this reason, and learning the action value Q associated with a state, action pair is a regression problem. So unless you have good reason to do otherwise, use a linear activation function on the output layer.</p>
  </li>
  <li><p>Control problems in continuous action spaces cannot be solved using only a learned action value function. That is because, in order to derive a greedy policy from a Q function, you need to solve <span class="math-container">$\pi(s) = \text{argmax}_aQ(s,a)$</span> and finding the maximum value in a continuous action space is not practical. If your Q network is part of an agent solving a control problem - i.e. finding the optimal policy - then you will need some other way to generate and improve the policy. This is usually done using a <a href="https://lilianweng.github.io/lil-log/2018/04/08/policy-gradient-algorithms.html" rel="nofollow noreferrer">policy gradient method</a>.</p>
  </li>
  </ul>
  <p>Due to this last point, it is sometimes advisable to discretise the action space if you want to use a simpler value-based RL method, such as DQN.</p>
  <hr />
  <p>* Caveats:</p>
  <ul>
  <li><p>There are a few architectures which really do use discrete inputs, weights and/or outputs. These are either historical variants or specialist though, you will not find any mainstream libraries that do this by default.</p>
  </li>
  <li><p>Computers model real numbers as <em>floating point</em> which are technically discrete due to practical limitations of CPUs. Sometimes it is important to know this, but it will not impact modelling a Q function.</p>
  </li>
  </ul>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-08-23T14:05:42.833Z
Answers.Text: Thanks Neil for your great answer (I had alreay upvoted and accepted it). One follow up question to your comment "Due to this last point, it is sometimes advisable to discretise the action space if you want to use a simpler value-based RL method, such as DQN."--> Basically I do want to use DQN (maybe also SARSA afterwards) and i would like to know whether it is also advisable to discretize the state space and not only the action space. My state space is quite large. I have about 10 variables that are all continious from min 0 to max 10,000.
Answers.UserDisplayName: PeterBe
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-08-23T14:09:38.080Z
Answers.Text: @PeterBe You do not need to discretise the state space for DQN. Do remember to scale the state values for use with a neural network (e.g. scale each value from -1.5 to 1.5 is reasonable)
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-08-23T14:27:08.127Z
Answers.Text: Thanks Neil for your anwer. I appreciate it. By scaling you mean to normalize or standardize each variable, right?
Answers.UserDisplayName: PeterBe
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-08-23T14:50:40.787Z
Answers.Text: @PeterBe Yes, you should scale to fit each vraiable into a standard range
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-08-23T16:36:12.683Z
Answers.Text: Hi Neil, thanks for your answers. I appreciate it. Basically I don't really understand why I should not discretize the state space but the action space. The ANN just maps inputs to outputs. When used in a RL environment the ANN maps state/action pairs to a reward as far as I understood. Surely, if you have less state/action pairs (which you will have if you dicretize the state space) the mapping of the ANN is not as challenging which will most propably lead to better results for the Q-learning algorithm
Answers.UserDisplayName: PeterBe
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-08-23T16:38:47.243Z
Answers.Text: @PeterBe: My answer spells that out. The discretisation is required to support max and argmax operations over the action space. You don't need to do that over the state space. Nothing stops you if you want to though.
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-08-16T13:39:04.990Z
Answers.LastActivityDate: 2021-08-18T10:39:47.663Z
Answers.LastEditDate: 2021-08-18T10:39:47.663Z
Answers.UserDisplayName: Neil Slater
Body: <p>In general, can ANNs have continuous inputs and outputs, or do they have to be discrete?</p>
  <p>So, basically, I would like to have a mapping of continuous inputs to continuous outputs. Is this possible? Does this depend on the type of ANN?</p>
  <p>More specifically, I would like to use neural networks for learning the Q-function in Reinforcement Learning. My problem has basically a continuous state and action space, and I would like to know whether I have to discretize it or not (a more detailed description of my problem can be found <a href="https://ai.stackexchange.com/q/28888/2444">here</a>).</p>
  
Tags: <neural-networks><reinforcement-learning><continuous-action-spaces><continuous-state-spaces><discretization>
Title: Can neural networks have continuous inputs and outputs, or do they have to be discrete?
UserDisplayName: PeterBe
ThreadText: # Can neural networks have continuous inputs and outputs, or do they have to be discrete?
  
  (Asked by: username_0 on 2021-08-16)
  
  In general, can ANNs have continuous inputs and outputs, or do they have to be discrete?
  
  So, basically, I would like to have a mapping of continuous inputs to continuous outputs. Is this possible? Does this depend on the type of ANN?
  
  More specifically, I would like to use neural networks for learning the Q-function in Reinforcement Learning. My problem has basically a continuous state and action space, and I would like to know whether I have to discretize it or not (a more detailed description of my problem can be found [here](https://ai.stackexchange.com/q/28888/2444)).
  
  **Tags:** neural-networks, reinforcement-learning, continuous-action-spaces, continuous-state-spaces, discretization
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2021-08-16. Score: 7 | Accepted answer
  
  Neural networks normally work in continuous spaces. A typical neural network function could be written as $f(\mathbf{x}, \mathbf{\theta}): \mathbb{R}^N \rightarrow \mathbb{R}^M$. That is, a function of some $N$ dimensional input vector of real numbers $\mathbf{x}$ that outputs some $M$ dimensional output vector of real numbers (which you could call $\mathbf{y}$) and that is parametrised by a vector of weights and biases $\mathbf{\theta}$.
  
  When you train a neural network with some data $\mathbf{X}, \mathbf{Y}$ then you are trying to adjust $\mathbf{\theta}$ so that $f(\mathbf{x}, \mathbf{\theta})$ approximates some assumed mapping function $g(\mathbf{x}): \mathcal{X} \rightarrow \mathcal{Y}$ that provides a "true" mapping between all possible inputs $\mathcal{X}$ and the matching correct values from all possible outputs $\mathcal{Y}$. In a neural network, then $\mathcal{X}$ and $\mathcal{Y}$ must be sets of vectors of real numbers.
  
  So, yes all neural networks map between continuous values by default\*. When NNs are used to work with discrete values, then the discrete variables need to be converted to real-valued variables in order to be usable with a NN. So discrete inputs might be mapped to a small set of different real values or a vector of $\{0, 1 \}$ (called "one hot" encoding). Discrete outputs are often mapped to a probability vector giving the confidence that the learned function assigns to each possible discrete output - this is the usual case in classifiers.
  
  You have a specific use case in mind:
  
  > More specifically, I would like to use neural networks for learning the Q-function in Reinforcement Learning. My problem has basically a continuous state and action space, and I would like to know whether I have to discretize it or not
  
  For an action value, or Q function in reinforcement learning, then you do not need to be concerned that state space or action space are continuous. However, there are some details that may be important:
  
  * Neural networks learn most efficiently when input elements are within a unit-like distribution such as $\mathcal{N}(0,1)$ (normal distribution with mean $0$, standard deviation $1$). This does not have to be precise, but you should take care to scale the input values so that each element has a typical magnitude around $1$.
  * The output values you want to learn need to be within the domain of the output layer's activation function. It is common to use linear (or "no" activation function) in the output layer for regression problems for this reason, and learning the action value Q associated with a state, action pair is a regression problem. So unless you have good reason to do otherwise, use a linear activation function on the output layer.
  * Control problems in continuous action spaces cannot be solved using only a learned action value function. That is because, in order to derive a greedy policy from a Q function, you need to solve $\pi(s) = \text{argmax}\_aQ(s,a)$ and finding the maximum value in a continuous action space is not practical. If your Q network is part of an agent solving a control problem - i.e. finding the optimal policy - then you will need some other way to generate and improve the policy. This is usually done using a [policy gradient method](https://lilianweng.github.io/lil-log/2018/04/08/policy-gradient-algorithms.html).
  
  Due to this last point, it is sometimes advisable to discretise the action space if you want to use a simpler value-based RL method, such as DQN.
  
  ---
  
  \* Caveats:
  
  * There are a few architectures which really do use discrete inputs, weights and/or outputs. These are either historical variants or specialist though, you will not find any mainstream libraries that do this by default.
  * Computers model real numbers as *floating point* which are technically discrete due to practical limitations of CPUs. Sometimes it is important to know this, but it will not impact modelling a Q function.
  
  > **Comments:**
  > **username_0**: Thanks Neil for your great answer (I had alreay upvoted and accepted it). One follow up question to your comment "Due to this last point, it is sometimes advisable to discretise the action space if you want to use a simpler value-based RL method, such as DQN."--> Basically I do want to use DQN (maybe also SARSA afterwards) and i would like to know whether it is also advisable to discretize the state space and not only the action space. My state space is quite large. I have about 10 variables that are all continious from min 0 to max 10,000.  
  > **username_1**: @PeterBe You do not need to discretise the state space for DQN. Do remember to scale the state values for use with a neural network (e.g. scale each value from -1.5 to 1.5 is reasonable)  
  > **username_0**: Thanks Neil for your anwer. I appreciate it. By scaling you mean to normalize or standardize each variable, right?  
  > **username_1**: @PeterBe Yes, you should scale to fit each vraiable into a standard range  
  > **username_0**: Hi Neil, thanks for your answers. I appreciate it. Basically I don't really understand why I should not discretize the state space but the action space. The ANN just maps inputs to outputs. When used in a RL environment the ANN maps state/action pairs to a reward as far as I understood. Surely, if you have less state/action pairs (which you will have if you dicretize the state space) the mapping of the ANN is not as challenging which will most propably lead to better results for the Q-learning algorithm  
  > **username_1**: @PeterBe: My answer spells that out. The discretisation is required to support max and argmax operations over the action space. You don't need to do that over the state space. Nothing stops you if you want to though.  
  
  ---
  
  