Answers.Body: <p>There is a much simpler answer than the ones already given. The value function represents the expected (discounted) returns from the current state when following policy <span class="math-container">$\pi$</span>, i.e:
  <span class="math-container">$$v_\pi(s) = \mathbb{E}\left[\sum_{k=0}^\infty \gamma^k R_{t+k} | S_t = s\right]\;.$$</span>
  This is just a scalar value that we would like to predict given some input, i.e. the state. So, predicting this quantity can be thought of (loosely) as a regression task. In regression, we typically want to learn a function <span class="math-container">$\hat{f}$</span> which, when given some input <span class="math-container">$x$</span>, can approximate well the output <span class="math-container">$y$</span> under the true (but unknown) function <span class="math-container">$f$</span>, i.e. we want to learn <span class="math-container">$\hat{f}$</span> such that <span class="math-container">$\hat{f}(x) \approx f(x)$</span>.</p>
  <p>Now, supposing our state space is too large (e.g. infinite) to be enumerated in a lookup table, we need to parameterise our approximate value function. This can be done using any parametrisable function <span class="math-container">$\hat{v}_\pi(s; \textbf{w})$</span>, where <span class="math-container">$\textbf{w}$</span> are the learnable parameters of the function, to approximate <span class="math-container">$v_\pi(s)$</span>. We choose neural networks because they have been successful empirically at approximating the true value function.</p>
  <p>To answer your question as to whether these are the best approximators for the value function, my answer would be currently yes, <em>but</em> there is a huge amount of research in the field of Reinforcement Learning and it is impossible to say whether or not they will remain state of the art.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-11-05T18:57:16.600Z
Answers.LastActivityDate: 2022-11-05T18:57:16.600Z
Answers.UserDisplayName: David
Answers.Body: <p>The <em>goodness</em> of the state in the <a href="https://github.com/ray-project/ray/blob/master/rllib/models/torch/fcnet.py#L126" rel="nofollow noreferrer">network</a> is determined by the <strong><code>activation_fn</code></strong>. The primary role of the Activation Function is to transform the summed weighted input from the node into an output value to be fed to the next hidden layer or as output. That adds non-linearity to the neural network, without that learning would be impossible. It allows the model to create complex mappings between the network’s inputs and outputs.</p>
  <p>The activation function used in hidden layers is typically chosen based on the type of neural network architecture. You need to match your activation function for your output layer based on the type of prediction problem that you are solving—specifically, the type of predicted variable.</p>
  <p>Also see the update, halfway through this answer.</p>
  <p>That can be set as follows:</p>
  <blockquote>
  <h1><a href="https://github.com/ray-project/ray/blob/master/rllib/models/catalog.py#L82" rel="nofollow noreferrer">Activation</a> function descriptor.</h1>
  <h1>Supported values are: &quot;tanh&quot;, &quot;relu&quot;, &quot;swish&quot; (or &quot;silu&quot;), &quot;linear&quot; (or None).</h1>
  <p>&quot;fcnet_activation&quot;: &quot;tanh&quot;,</p>
  </blockquote>
  <p>With a linear transfer function the value would look like this:</p>
  <p><a href="https://i.sstatic.net/VcBz0.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/VcBz0.png" alt="enter image description here" /></a></p>
  <p>Fig: Linear Activation Function - Equation: f(x) = x</p>
  <p>It doesn’t help with the complexity or various parameters of usual data that is fed to the neural networks. All layers of the neural network will collapse into one if a linear activation function is used. No matter the number of layers in the neural network, the last layer will still be a linear function of the first layer. So, essentially, a linear activation function turns the neural network into just one layer.</p>
  <p>The Nonlinear Activation Functions are the most used activation functions.</p>
  <ol>
  <li>Sigmoid or Logistic Activation Function</li>
  </ol>
  <p>The Sigmoid Function curve looks like a S-shape.</p>
  <p><a href="https://i.sstatic.net/jETqK.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/jETqK.png" alt="enter image description here" /></a><br />
  Fig: Sigmoid Function</p>
  <p>The main reason why we use sigmoid function is because its valued between 0 to 1. It's used for models where we have to predict the probability as an output.</p>
  <p>Since probability of anything exists only between the range of 0 and 1, the sigmoid is a useful function and is differentiable. That means, we can find the slope of the sigmoid curve at any two points.</p>
  <p>The function is monotonic but the function’s derivative is not. The logistic sigmoid function can cause a neural network to get stuck at the training time.</p>
  <ol start="2">
  <li>Tanh or hyperbolic tangent Activation Function<br />
  The tanh is also like a logistic sigmoid but better. The range of the tanh function is from -1 to 1. tanh is also sigmoidal (s-shaped).</li>
  </ol>
  <p><a href="https://i.sstatic.net/ai9Vo.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/ai9Vo.png" alt="enter image description here" /></a><br />
  Fig: tanh v/s Logistic Sigmoid<br />
  The advantage is that the negative inputs will be mapped strongly negative and the zero inputs will be mapped near zero in the tanh graph.</p>
  <p>The function is differentiable and monotonic, while its derivative is not monotonic.</p>
  <p>The tanh function is mainly used for classification between two classes.</p>
  <p>Both tanh and the logistic sigmoid activation functions are used in feed-forward nets.</p>
  <ol start="3">
  <li>ReLU (Rectified Linear Unit) Activation Function<br />
  The ReLU is the most used activation function in the world right now, since it's used in almost all the convolutional neural networks or deep learning.</li>
  </ol>
  <p><a href="https://i.sstatic.net/shfHZ.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/shfHZ.png" alt="enter image description here" /></a><br />
  Fig: ReLU v/s Logistic Sigmoid<br />
  The ReLU is half rectified (from bottom), f(z) is zero when z is less than zero and f(z) is equal to z when z is above or equal to zero, with a range of 0 to infinity.</p>
  <p>The function and its derivative both are monotonic, but the issue is that all the negative values become zero immediately; which decreases the ability of the model to fit or train from the data properly.
  That means any negative input given to the ReLU activation function turns the value into zero immediately in the graph, which in turns affects the resulting graph by not mapping the negative values appropriately.</p>
  <ol start="4">
  <li>Leaky ReLU<br />
  It's an attempt to solve the dying ReLU problem.</li>
  </ol>
  <p><a href="https://i.sstatic.net/vflYB.jpg" rel="nofollow noreferrer"><img src="https://i.sstatic.net/vflYB.jpg" alt="enter image description here" /></a><br />
  Fig: ReLU v/s Leaky ReLU</p>
  <p>The leak helps to increase the range of the ReLU function. Usually, the value of <span class="math-container">$\LARGE{a}$</span> is 0.01 or so.</p>
  <p>When a is not 0.01 then it is called Randomized ReLU.</p>
  <p>The range of the Leaky ReLU is -infinity to infinity.</p>
  <p>Both Leaky and Randomized ReLU functions are monotonic in nature, and their derivatives are also monotonic in nature.</p>
  <p><strong>What is a Good Activation Function?</strong><br />
  A proper choice has to be made in choosing the activation function to improve the results in neural network computing. All activation functions must be monotonic, differentiable, and quickly converging with respect to the weights for optimization purposes.</p>
  <p><strong>Why derivative/differentiation is used ?</strong><br />
  When updating the curve, to know in which direction and how much to change or update the curve depending upon the slope, we use differentiation.</p>
  <p>It makes it easy for the model to generalize or adapt with variety of data and to differentiate between the output.</p>
  <p><a href="https://i.sstatic.net/z5xnE.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/z5xnE.png" alt="enter image description here" /></a></p>
  <p>You can learn more about differentiation at these blogs: <a href="https://www.analyticsvidhya.com/blog/2019/10/mathematics-behind-machine-learning/" rel="nofollow noreferrer">&quot;Mathematics behind Machine Learning – The Core Concepts you Need to Know&quot;</a> and <a href="https://machinelearningmastery.com/application-of-differentiations-in-neural-networks/" rel="nofollow noreferrer">&quot;Application of differentiations in neural networks&quot;</a>.</p>
  <p>An excellent resource for activation functions is: <a href="https://itnext.io/ultimate-guide-to-activation-functions-37740be5942" rel="nofollow noreferrer">&quot;Ultimate Guide to Activation Functions&quot;</a>.</p>
  <blockquote>
  <p>&quot;Is the neural network value function the best method of achieving such a &quot;goodness&quot; value in an environment with many possible states and indefinite length?&quot;.</p>
  </blockquote>
  <p>Yes, a properly trained neural network can not only answer what it was trained for but can also infer additional questions and provide correct answers from a relatively compact representation.</p>
  <p>The optimization algorithm is what carries out the learning process in a neural network. There are many different optimization algorithms. They are different in terms of memory requirements, processing speed, and numerical precision</p>
  <p>The <strong>loss index</strong> plays a vital role in the use of neural networks. It defines the task the neural network is required to do and provides a <strong>measure of the quality</strong> (goodness) of the representation it is required to learn. The choice of a suitable loss index depends on the application.</p>
  <p>When setting a loss index, two different terms must be chosen: an error term and a regularization term.</p>
  <p><span class="math-container">$$\text{loss_index}=\text{error_term}+\text{regularization_term}$$</span></p>
  <p>The <strong>error term</strong> is the most important term in the loss expression. It measures how the neural network fits the data set.</p>
  <p>All those errors can be measured over different subsets of the data. In this regard, the training error refers to the error measured on the training samples, the selection error is measured on the selection samples, and the testing error is measured on the testing samples.</p>
  <p>A solution is regular when small changes in the input variables led to small changes in the outputs. An approach for non-regular problems is to control the effective complexity of the neural network. We can achieve this by including a <strong>regularization term</strong> in the loss index.</p>
  <p>Regularization terms usually measure the values of the parameters in the neural network. Adding that term to the error will cause the neural network to have smaller weights and biases, which will force its response to be smoother (with a greater goodness of fit).</p>
  <blockquote>
  <p>&quot;The learning problem is formulated as the minimization of a loss index, <span class="math-container">$\LARGE{f}$</span>. It is a function that measures the performance of a neural network on a data set.</p>
  </blockquote>
  <blockquote>
  <p>The loss index includes, in general, an error and a regularization terms. The error term evaluates how a neural network fits the data set. The regularization term prevents overfitting by controlling the model's complexity.</p>
  </blockquote>
  <blockquote>
  <p>The loss function depends on the adaptative parameters (biases and synaptic weights) in the neural network. We can group them into a single n-dimensional weight vector <span class="math-container">$\LARGE{w}$</span>.&quot;.</p>
  </blockquote>
  <p>For further details see: <a href="https://www.neuraldesigner.com/blog/5_algorithms_to_train_a_neural_network" rel="nofollow noreferrer">&quot;5 algorithms to train a neural network&quot;</a>.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-11-04T23:21:59.610Z
Answers.Text: Thank you for posting an answer. I have taken the time to read your answer and the provided links at the bottom. I still fail to see how a network represents the "goodness" of a given state in reinforcement learning. In your answer you say that it is the activation_fn which determines the "goodness" of a state, but I don't see why that is. Can you explain why "the goodness of the state in the network is determined by the activation_fn"?
Answers.UserDisplayName: MrDrago9
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-12-20T15:10:31.417Z
Answers.Text: Goodness is not determined by the activation function, although activation functions certainly play a role on the neural network's output. The goodness is determined by a loss or objective function. This answer is too long and focus too much on the activation functions, which are not really the key to answer the question.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-12-20T18:23:25.333Z
Answers.Text: @nbro, you've mis-read the question and didn't read the code  The question is: "... the network in [***this*** example](https://github.com/ray-project/ray/blob/master/rllib/models/torch/fcnet.py#L126) , represent how good ...", and the code for [slimFC](https://github.com/ray-project/ray/blob/1e34aa78c7550b1f9631e46902e4cca89093557b/rllib/models/torch/misc.py#L256) is called to use a linear function without an activation layer. It's not a different question than what they asked, don't edit away the question and downvote answers. Answer the question **as asked**. This is your 3rd action here.
Answers.UserDisplayName: Rob
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-12-20T20:52:01.550Z
Answers.Text: @Rob I didn't misread the question. The question is, how can a neural network represent how good it's to be in a state? The other question is why are neural networks used as value functions in RL? You went on to explain things about activation functions, which are not very relevant to answer these questions. The user wrote "like this" and he didn't even mention activation functions anywhere. I don't think that the point was about the activation functions of the specific NN. I think the point was that it was unclear to them how a neural network can represent $v(s)$ or $q(s, a)$.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-12-20T20:54:15.447Z
Answers.Text: Sure, activation functions are a part of the explanation of why neural networks can approximate, compute or are functions, but your answer goes too much in detail of activation functions and I don't think that was the key. And your answer is simply wrong. The goodness of a state is not determined by the activation function alone, but it's what the neural network outputs. The goodness of state is a $v(s)$ and that's what a neural network can output.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-11-04T08:54:43.330Z
Answers.LastActivityDate: 2022-11-05T17:21:54.010Z
Answers.LastEditDate: 2022-11-05T17:21:54.010Z
Answers.UserDisplayName: Rob
Answers.Body: <p>Informally I would agree with your understanding of the value function.</p>
  <p>To answer your questions, it seems explaining some motivation for parametric modeling/machine learning is necessary: In general if we are not in a tabular setting, we wish to take advantage of the nature of our (continuous) data by leveraging relationships between states and rewards. In other words, we may think that there exists some underlying pattern in the states that relates to how much reward we can attain. More formally, one may posit some sort of parametric model, which implicitly assumes there is some parameter vector <span class="math-container">$\theta_0$</span> s.t. <span class="math-container">$V(s)=\theta_0' s, \forall s \in S$</span>. Or similarly for the Q value function.</p>
  <p>This kind of assumption is similar to any linear regression model, or just any model that wants to take advantage of potential underlying patterns in the data to learn (like in any other machine learning problem). The choice of a neural network is motivated well by the amount of data available in certain domain areas and the recent advent of empirical success of neural nets.</p>
  <p>With this setup, to more specifically answer your last two questions: the network represents this value because we model it to be that way as we model <span class="math-container">$y=\theta'x$</span> in linear regression. Is this the best method? So far, this is one of the best methods we have available <em>provided we have enough data</em>. The reason can be supported by empirical success seen in other applications, and mathematically by results like the universal approximation theorem (which states that any continuous function can be represented arbitrarily well by a NN), or other more modern results from high dimensional statistics.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-12-20T15:16:38.397Z
Answers.Text: I am not sure why this question was downvoted. I am trying to see where the big problem is.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-11-05T02:50:14.187Z
Answers.LastActivityDate: 2022-11-05T02:55:38.090Z
Answers.LastEditDate: 2022-11-05T02:55:38.090Z
Answers.UserDisplayName: sma
Body: <p>My understanding is that a value function in reinforcement learning returns a value that represents how &quot;good&quot; it is to be in a given state.</p>
  <p>How does a network, such as the network in <a href="https://github.com/ray-project/ray/blob/master/rllib/models/torch/fcnet.py#L126" rel="nofollow noreferrer">this example </a>, represent how good it is to be in the given state?</p>
  <p>Is the neural network value function the best method of achieving such a &quot;goodness&quot; value in an environment with many possible states and indefinite length?</p>
  
Tags: <neural-networks><reinforcement-learning><deep-rl><value-functions>
Title: Why are neural networks used as reinforcement learning model value functions?
UserDisplayName: MrDrago9
ThreadText: # Why are neural networks used as reinforcement learning model value functions?
  
  (Asked by: username_0 on 2022-11-01)
  
  My understanding is that a value function in reinforcement learning returns a value that represents how "good" it is to be in a given state.
  
  How does a network, such as the network in [this example](https://github.com/ray-project/ray/blob/master/rllib/models/torch/fcnet.py#L126) , represent how good it is to be in the given state?
  
  Is the neural network value function the best method of achieving such a "goodness" value in an environment with many possible states and indefinite length?
  
  **Tags:** neural-networks, reinforcement-learning, deep-rl, value-functions
  
  ---
  
  ## 3 answers
  
  ### Answer by username_1 on 2022-11-05. Score: 0 | Accepted answer
  
  There is a much simpler answer than the ones already given. The value function represents the expected (discounted) returns from the current state when following policy $\pi$, i.e:
  $$v\_\pi(s) = \mathbb{E}\left[\sum\_{k=0}^\infty \gamma^k R\_{t+k} | S\_t = s\right]\;.$$
  This is just a scalar value that we would like to predict given some input, i.e. the state. So, predicting this quantity can be thought of (loosely) as a regression task. In regression, we typically want to learn a function $\hat{f}$ which, when given some input $x$, can approximate well the output $y$ under the true (but unknown) function $f$, i.e. we want to learn $\hat{f}$ such that $\hat{f}(x) \approx f(x)$.
  
  Now, supposing our state space is too large (e.g. infinite) to be enumerated in a lookup table, we need to parameterise our approximate value function. This can be done using any parametrisable function $\hat{v}\_\pi(s; \textbf{w})$, where $\textbf{w}$ are the learnable parameters of the function, to approximate $v\_\pi(s)$. We choose neural networks because they have been successful empirically at approximating the true value function.
  
  To answer your question as to whether these are the best approximators for the value function, my answer would be currently yes, *but* there is a huge amount of research in the field of Reinforcement Learning and it is impossible to say whether or not they will remain state of the art.
  
  
  ---
  
  ### Answer by username_2 on 2022-11-04. Score: -2
  
  The *goodness* of the state in the [network](https://github.com/ray-project/ray/blob/master/rllib/models/torch/fcnet.py#L126) is determined by the **`activation_fn`**. The primary role of the Activation Function is to transform the summed weighted input from the node into an output value to be fed to the next hidden layer or as output. That adds non-linearity to the neural network, without that learning would be impossible. It allows the model to create complex mappings between the network’s inputs and outputs.
  
  The activation function used in hidden layers is typically chosen based on the type of neural network architecture. You need to match your activation function for your output layer based on the type of prediction problem that you are solving—specifically, the type of predicted variable.
  
  Also see the update, halfway through this answer.
  
  That can be set as follows:
  
  > # [Activation](https://github.com/ray-project/ray/blob/master/rllib/models/catalog.py#L82) function descriptor.
  >
  > # Supported values are: "tanh", "relu", "swish" (or "silu"), "linear" (or None).
  >
  > "fcnet\_activation": "tanh",
  
  With a linear transfer function the value would look like this:
  
  [![enter image description here](https://i.sstatic.net/VcBz0.png)](https://i.sstatic.net/VcBz0.png)
  
  Fig: Linear Activation Function - Equation: f(x) = x
  
  It doesn’t help with the complexity or various parameters of usual data that is fed to the neural networks. All layers of the neural network will collapse into one if a linear activation function is used. No matter the number of layers in the neural network, the last layer will still be a linear function of the first layer. So, essentially, a linear activation function turns the neural network into just one layer.
  
  The Nonlinear Activation Functions are the most used activation functions.
  
  1. Sigmoid or Logistic Activation Function
  
  The Sigmoid Function curve looks like a S-shape.
  
  [![enter image description here](https://i.sstatic.net/jETqK.png)](https://i.sstatic.net/jETqK.png)  
  Fig: Sigmoid Function
  
  The main reason why we use sigmoid function is because its valued between 0 to 1. It's used for models where we have to predict the probability as an output.
  
  Since probability of anything exists only between the range of 0 and 1, the sigmoid is a useful function and is differentiable. That means, we can find the slope of the sigmoid curve at any two points.
  
  The function is monotonic but the function’s derivative is not. The logistic sigmoid function can cause a neural network to get stuck at the training time.
  
  2. Tanh or hyperbolic tangent Activation Function  
     The tanh is also like a logistic sigmoid but better. The range of the tanh function is from -1 to 1. tanh is also sigmoidal (s-shaped).
  
  [![enter image description here](https://i.sstatic.net/ai9Vo.png)](https://i.sstatic.net/ai9Vo.png)  
  Fig: tanh v/s Logistic Sigmoid  
  The advantage is that the negative inputs will be mapped strongly negative and the zero inputs will be mapped near zero in the tanh graph.
  
  The function is differentiable and monotonic, while its derivative is not monotonic.
  
  The tanh function is mainly used for classification between two classes.
  
  Both tanh and the logistic sigmoid activation functions are used in feed-forward nets.
  
  3. ReLU (Rectified Linear Unit) Activation Function  
     The ReLU is the most used activation function in the world right now, since it's used in almost all the convolutional neural networks or deep learning.
  
  [![enter image description here](https://i.sstatic.net/shfHZ.png)](https://i.sstatic.net/shfHZ.png)  
  Fig: ReLU v/s Logistic Sigmoid  
  The ReLU is half rectified (from bottom), f(z) is zero when z is less than zero and f(z) is equal to z when z is above or equal to zero, with a range of 0 to infinity.
  
  The function and its derivative both are monotonic, but the issue is that all the negative values become zero immediately; which decreases the ability of the model to fit or train from the data properly.
  That means any negative input given to the ReLU activation function turns the value into zero immediately in the graph, which in turns affects the resulting graph by not mapping the negative values appropriately.
  
  4. Leaky ReLU  
     It's an attempt to solve the dying ReLU problem.
  
  [![enter image description here](https://i.sstatic.net/vflYB.jpg)](https://i.sstatic.net/vflYB.jpg)  
  Fig: ReLU v/s Leaky ReLU
  
  The leak helps to increase the range of the ReLU function. Usually, the value of $\LARGE{a}$ is 0.01 or so.
  
  When a is not 0.01 then it is called Randomized ReLU.
  
  The range of the Leaky ReLU is -infinity to infinity.
  
  Both Leaky and Randomized ReLU functions are monotonic in nature, and their derivatives are also monotonic in nature.
  
  **What is a Good Activation Function?**  
  A proper choice has to be made in choosing the activation function to improve the results in neural network computing. All activation functions must be monotonic, differentiable, and quickly converging with respect to the weights for optimization purposes.
  
  **Why derivative/differentiation is used ?**  
  When updating the curve, to know in which direction and how much to change or update the curve depending upon the slope, we use differentiation.
  
  It makes it easy for the model to generalize or adapt with variety of data and to differentiate between the output.
  
  [![enter image description here](https://i.sstatic.net/z5xnE.png)](https://i.sstatic.net/z5xnE.png)
  
  You can learn more about differentiation at these blogs: ["Mathematics behind Machine Learning – The Core Concepts you Need to Know"](https://www.analyticsvidhya.com/blog/2019/10/mathematics-behind-machine-learning/) and ["Application of differentiations in neural networks"](https://machinelearningmastery.com/application-of-differentiations-in-neural-networks/).
  
  An excellent resource for activation functions is: ["Ultimate Guide to Activation Functions"](https://itnext.io/ultimate-guide-to-activation-functions-37740be5942).
  
  > "Is the neural network value function the best method of achieving such a "goodness" value in an environment with many possible states and indefinite length?".
  
  Yes, a properly trained neural network can not only answer what it was trained for but can also infer additional questions and provide correct answers from a relatively compact representation.
  
  The optimization algorithm is what carries out the learning process in a neural network. There are many different optimization algorithms. They are different in terms of memory requirements, processing speed, and numerical precision
  
  The **loss index** plays a vital role in the use of neural networks. It defines the task the neural network is required to do and provides a **measure of the quality** (goodness) of the representation it is required to learn. The choice of a suitable loss index depends on the application.
  
  When setting a loss index, two different terms must be chosen: an error term and a regularization term.
  
  $$\text{loss\_index}=\text{error\_term}+\text{regularization\_term}$$
  
  The **error term** is the most important term in the loss expression. It measures how the neural network fits the data set.
  
  All those errors can be measured over different subsets of the data. In this regard, the training error refers to the error measured on the training samples, the selection error is measured on the selection samples, and the testing error is measured on the testing samples.
  
  A solution is regular when small changes in the input variables led to small changes in the outputs. An approach for non-regular problems is to control the effective complexity of the neural network. We can achieve this by including a **regularization term** in the loss index.
  
  Regularization terms usually measure the values of the parameters in the neural network. Adding that term to the error will cause the neural network to have smaller weights and biases, which will force its response to be smoother (with a greater goodness of fit).
  
  > "The learning problem is formulated as the minimization of a loss index, $\LARGE{f}$. It is a function that measures the performance of a neural network on a data set.
  
  > The loss index includes, in general, an error and a regularization terms. The error term evaluates how a neural network fits the data set. The regularization term prevents overfitting by controlling the model's complexity.
  
  > The loss function depends on the adaptative parameters (biases and synaptic weights) in the neural network. We can group them into a single n-dimensional weight vector $\LARGE{w}$.".
  
  For further details see: ["5 algorithms to train a neural network"](https://www.neuraldesigner.com/blog/5_algorithms_to_train_a_neural_network).
  
  > **Comments:**
  > **username_0**: Thank you for posting an answer. I have taken the time to read your answer and the provided links at the bottom. I still fail to see how a network represents the "goodness" of a given state in reinforcement learning. In your answer you say that it is the activation_fn which determines the "goodness" of a state, but I don't see why that is. Can you explain why "the goodness of the state in the network is determined by the activation_fn"?  
  > **username_3**: Goodness is not determined by the activation function, although activation functions certainly play a role on the neural network's output. The goodness is determined by a loss or objective function. This answer is too long and focus too much on the activation functions, which are not really the key to answer the question.  
  > **username_2**: @nbro, you've mis-read the question and didn't read the code  The question is: "... the network in [***this*** example](https://github.com/ray-project/ray/blob/master/rllib/models/torch/fcnet.py#L126) , represent how good ...", and the code for [slimFC](https://github.com/ray-project/ray/blob/1e34aa78c7550b1f9631e46902e4cca89093557b/rllib/models/torch/misc.py#L256) is called to use a linear function without an activation layer. It's not a different question than what they asked, don't edit away the question and downvote answers. Answer the question **as asked**. This is your 3rd action here.  
  > **username_3**: @Rob I didn't misread the question. The question is, how can a neural network represent how good it's to be in a state? The other question is why are neural networks used as value functions in RL? You went on to explain things about activation functions, which are not very relevant to answer these questions. The user wrote "like this" and he didn't even mention activation functions anywhere. I don't think that the point was about the activation functions of the specific NN. I think the point was that it was unclear to them how a neural network can represent $v(s)$ or $q(s, a)$.  
  > **username_3**: Sure, activation functions are a part of the explanation of why neural networks can approximate, compute or are functions, but your answer goes too much in detail of activation functions and I don't think that was the key. And your answer is simply wrong. The goodness of a state is not determined by the activation function alone, but it's what the neural network outputs. The goodness of state is a $v(s)$ and that's what a neural network can output.  
  
  ---
  
  ### Answer by username_4 on 2022-11-05. Score: -2
  
  Informally I would agree with your understanding of the value function.
  
  To answer your questions, it seems explaining some motivation for parametric modeling/machine learning is necessary: In general if we are not in a tabular setting, we wish to take advantage of the nature of our (continuous) data by leveraging relationships between states and rewards. In other words, we may think that there exists some underlying pattern in the states that relates to how much reward we can attain. More formally, one may posit some sort of parametric model, which implicitly assumes there is some parameter vector $\theta\_0$ s.t. $V(s)=\theta\_0' s, \forall s \in S$. Or similarly for the Q value function.
  
  This kind of assumption is similar to any linear regression model, or just any model that wants to take advantage of potential underlying patterns in the data to learn (like in any other machine learning problem). The choice of a neural network is motivated well by the amount of data available in certain domain areas and the recent advent of empirical success of neural nets.
  
  With this setup, to more specifically answer your last two questions: the network represents this value because we model it to be that way as we model $y=\theta'x$ in linear regression. Is this the best method? So far, this is one of the best methods we have available *provided we have enough data*. The reason can be supported by empirical success seen in other applications, and mathematically by results like the universal approximation theorem (which states that any continuous function can be represented arbitrarily well by a NN), or other more modern results from high dimensional statistics.
  
  > **Comments:**
  > **username_3**: I am not sure why this question was downvoted. I am trying to see where the big problem is.  
  
  ---
  
  