Answers.Body: <p>The biggest problem with SVMs, random forests, gradient boosting and others for reinforcement learning (RL) is that they are not able to learn online, adjusting for new data as it arrives, and equally importantly forgetting older data. These algorithms learn from fixed datasets, and must be re-trained with whole new datasets if the target dataset changes, in scenarios also known as non-stationary problems.</p>
  <p>This is a major issue in reinforcement learning for control problems (when you want to find an optimal policy). RL control problems always generate non-stationary target data, because they maintain a current policy - or for off-policy approches, current target policy. That current policy then changes over time as the agent improves its estimates (of e.g. the Q table), which means it is necessary to forget older training data and replace with newer.</p>
  <p>So one requirement in RL is for online learning algorithms that can handle non-stationary data well - both incorporating new data and smoothly forgetting older data. That includes neural networks. There are not many other non-linear approximators that can do so simply.</p>
  <p>It is possible to compensate for this lack of online learning in various ways. Some of the model classes, such as <a href="https://proceedings.neurips.cc/paper/2014/file/d1dc3a8270a6f9394f88847d7f0050cf-Paper.pdf" rel="nofollow noreferrer">random forests</a> and <a href="https://proceedings.neurips.cc/paper/2000/file/155fa09596c7e18e50b58eb7e0c6ccb4-Paper.pdf" rel="nofollow noreferrer">support vector machines</a>, can be extended to learn online - although this seems not to be explored much in deep RL, perhaps due to complexity, or limitations that make it still less flexible choice than neural networks (opinion: looking at the recent dates on these papers compared to the longer history of using NNs with RL, there may be an element of intertia too, and possibility for revisiting using these models in a RL context).</p>
  <p>You could modify the outer algorithm to be more like policy iteration, collecting large amounts of data for the current policy to form a large batch that is then learned from to replace the previous approximate value function and in turn update the policy from that. That would work for most approximators in theory, but may be very slow and introduces hyperparameter of batch size that will have a large impact.</p>
  <p>There are unwanted side-effect to online learning in RL using approximators, such as the risk of <a href="https://en.wikipedia.org/wiki/Catastrophic_interference" rel="nofollow noreferrer">catastrophic forgetting</a>, where a well-trained agent is not able to distinguish bad choices from good ones because it has not experienced a bad decision for many time steps, and has generalised that &quot;everything is good now&quot;. This would not be fixed in RL by using large batches and non-online algorithms - if anything it could be worse in those cases, because they would effectively be forced to forget everything between batches. The usual fix for catastrophic forgetting in RL is adding some management of the dataset - e.g. keeping some amount of older experience even though it is technically not part of the current policy's generated data.</p>
  <blockquote>
  <p>they also have a huge number of trainable parameters and are extremely prone to overfitting</p>
  </blockquote>
  <p>Overfitting of approximators can happen, but may be less of an issue in practice with RL compared to smaller fixed datasets, because agents are continuously and actively generating new data. Complex environments with lots of resources used to generate experience behave in a lot of ways like &quot;big data&quot; datasets, playing to the strengths of large flexible models.</p>
  <p>In RL there is commonly an overlap with processing image data (and possibly other sensors with return data that can be structured into some grid or graph). In the problem domains related to image comprehension, neural networks are state of the art, so it is no surprise to see them used for approximating action values in the orginal DQN paper for example - with the right architecture NNs are measurably superior, in terms of error metrics, to SVMs, RFs, boosting et al, when fitting this kind of sensor data.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-06-06T15:56:05.317Z
Answers.Text: Comments are not for extended discussion; this conversation has been [moved to chat](https://chat.stackexchange.com/rooms/136844/discussion-on-answer-by-neil-slater-why-are-only-neural-networks-and-not-svms).
Answers.UserDisplayName: Dennis Soemers
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-06-05T18:53:50.217Z
Answers.LastActivityDate: 2022-06-06T11:51:55.647Z
Answers.LastEditDate: 2022-06-06T11:51:55.647Z
Answers.UserDisplayName: Neil Slater
Body: <p>I know that neural networks are the &quot;universal function approximator&quot;, but they also have a huge number of trainable parameters and are extremely prone to overfitting.</p>
  <p>So my question is: Why aren't SVMs or Random Forests used as the mediators in decision-making?</p>
  <p>I have read <a href="https://ai.stackexchange.com/questions/6407/can-agents-be-implemented-with-ml-algorithms-other-than-neural-networks">this response</a>, and it highlights very broadly why NNs are not <em>required</em> but are generally good for the job.</p>
  <p>This does not answer my question, though, as I want to understand why no popular implementations (that I've seen) use anything but neural networks, and why the vast majority of reinforcement learning literature talks about Deep RL and not, for example, &quot;SVM-RL&quot; or something like that.</p>
  <p>My understanding is that this is technically possible - but why is it not being done often or at all?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-06-05T18:05:24.317Z
Comments.Text: Hi, I have linked this to a question that I think is a duplicate. I have a high rep on RL, so it closed it immediately instead of putting it to a vote. If you think there is something missing from the linked question and answer, please modify your question to make that obvious and comment. I'm happy to vote to reopen if there is some new angle on this question that you want answered. I also up-voted, because (natural) duplicates are a useful thing to help others searching
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-06-05T18:27:42.543Z
Comments.Text: @NeilSlater Thank you. I have modified the question to be more specific, let me know if you think that's good.
Comments.UserDisplayName: Vladimir Belik
Tags: <neural-networks><reinforcement-learning><function-approximation>
Title: Why are only neural networks (and not SVMs, for example) used for reinforcement learning?
UserDisplayName: Vladimir Belik
ThreadText: # Why are only neural networks (and not SVMs, for example) used for reinforcement learning?
  
  (Asked by: username_0 on 2022-06-05)
  
  I know that neural networks are the "universal function approximator", but they also have a huge number of trainable parameters and are extremely prone to overfitting.
  
  So my question is: Why aren't SVMs or Random Forests used as the mediators in decision-making?
  
  I have read [this response](https://ai.stackexchange.com/questions/6407/can-agents-be-implemented-with-ml-algorithms-other-than-neural-networks), and it highlights very broadly why NNs are not *required* but are generally good for the job.
  
  This does not answer my question, though, as I want to understand why no popular implementations (that I've seen) use anything but neural networks, and why the vast majority of reinforcement learning literature talks about Deep RL and not, for example, "SVM-RL" or something like that.
  
  My understanding is that this is technically possible - but why is it not being done often or at all?
  
  **Tags:** neural-networks, reinforcement-learning, function-approximation
  
  > **Comments:**
  > **username_1**: Hi, I have linked this to a question that I think is a duplicate. I have a high rep on RL, so it closed it immediately instead of putting it to a vote. If you think there is something missing from the linked question and answer, please modify your question to make that obvious and comment. I'm happy to vote to reopen if there is some new angle on this question that you want answered. I also up-voted, because (natural) duplicates are a useful thing to help others searching  
  > **username_0**: @username_1 Thank you. I have modified the question to be more specific, let me know if you think that's good.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2022-06-05. Score: 5 | Accepted answer
  
  The biggest problem with SVMs, random forests, gradient boosting and others for reinforcement learning (RL) is that they are not able to learn online, adjusting for new data as it arrives, and equally importantly forgetting older data. These algorithms learn from fixed datasets, and must be re-trained with whole new datasets if the target dataset changes, in scenarios also known as non-stationary problems.
  
  This is a major issue in reinforcement learning for control problems (when you want to find an optimal policy). RL control problems always generate non-stationary target data, because they maintain a current policy - or for off-policy approches, current target policy. That current policy then changes over time as the agent improves its estimates (of e.g. the Q table), which means it is necessary to forget older training data and replace with newer.
  
  So one requirement in RL is for online learning algorithms that can handle non-stationary data well - both incorporating new data and smoothly forgetting older data. That includes neural networks. There are not many other non-linear approximators that can do so simply.
  
  It is possible to compensate for this lack of online learning in various ways. Some of the model classes, such as [random forests](https://proceedings.neurips.cc/paper/2014/file/d1dc3a8270a6f9394f88847d7f0050cf-Paper.pdf) and [support vector machines](https://proceedings.neurips.cc/paper/2000/file/155fa09596c7e18e50b58eb7e0c6ccb4-Paper.pdf), can be extended to learn online - although this seems not to be explored much in deep RL, perhaps due to complexity, or limitations that make it still less flexible choice than neural networks (opinion: looking at the recent dates on these papers compared to the longer history of using NNs with RL, there may be an element of intertia too, and possibility for revisiting using these models in a RL context).
  
  You could modify the outer algorithm to be more like policy iteration, collecting large amounts of data for the current policy to form a large batch that is then learned from to replace the previous approximate value function and in turn update the policy from that. That would work for most approximators in theory, but may be very slow and introduces hyperparameter of batch size that will have a large impact.
  
  There are unwanted side-effect to online learning in RL using approximators, such as the risk of [catastrophic forgetting](https://en.wikipedia.org/wiki/Catastrophic_interference), where a well-trained agent is not able to distinguish bad choices from good ones because it has not experienced a bad decision for many time steps, and has generalised that "everything is good now". This would not be fixed in RL by using large batches and non-online algorithms - if anything it could be worse in those cases, because they would effectively be forced to forget everything between batches. The usual fix for catastrophic forgetting in RL is adding some management of the dataset - e.g. keeping some amount of older experience even though it is technically not part of the current policy's generated data.
  
  > they also have a huge number of trainable parameters and are extremely prone to overfitting
  
  Overfitting of approximators can happen, but may be less of an issue in practice with RL compared to smaller fixed datasets, because agents are continuously and actively generating new data. Complex environments with lots of resources used to generate experience behave in a lot of ways like "big data" datasets, playing to the strengths of large flexible models.
  
  In RL there is commonly an overlap with processing image data (and possibly other sensors with return data that can be structured into some grid or graph). In the problem domains related to image comprehension, neural networks are state of the art, so it is no surprise to see them used for approximating action values in the orginal DQN paper for example - with the right architecture NNs are measurably superior, in terms of error metrics, to SVMs, RFs, boosting et al, when fitting this kind of sensor data.
  
  > **Comments:**
  > **username_2**: Comments are not for extended discussion; this conversation has been [moved to chat](https://chat.stackexchange.com/rooms/136844/discussion-on-answer-by-username_1-why-are-only-neural-networks-and-not-svms).  
  
  ---
  
  