Answers.Body: <p>Even after several years of success of deep learning systems (i.e. neural networks trained with gradient descent and back-propagation), as far as I know, there is not yet a consensus on what constitutes a deep neural network. Some people could use a neural network with 2 hidden layers and call it deep (like in your case), but other people may just dedicate the adjective <em>deep</em> to refer to neural networks with 10, 100, or more hidden layers. In fact, there are some good reasons to associate the term <em>deep</em> only to neural networks that have a significant number of hidden layers (e.g. 100): for example, the exploding (or vanishing) gradient problem does not typically arise if you only have one hidden layer but can easily occur with many (e.g. 100) hidden layers.</p>
  <p>Nevertheless, a neural network with at least one hidden layer can approximate any continuous function, given enough (but finite number of) units (or neurons) in the layers. See the <a href="https://en.wikipedia.org/wiki/Universal_approximation_theorem" rel="nofollow noreferrer">universal approximation theorem</a>. For this reason, we could start denoting any such neural network as deep, but, although this rule would exclude perceptrons (which can only approximate linear functions, and nobody would probably call them deep anyway), this rule would be a bit redundant or useless (i.e. we may just not use the adjective <em>deep</em> to start with).</p>
  <p>In your case, the rule that the authors are using seems to be the following: if it contains more hidden layers than the bare minimum (i.e. 1) to approximate any continuous function, then let's denote it as <em>deep</em>.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-12-31T21:57:45.977Z
Answers.Text: That's a very good answer!
Answers.UserDisplayName: High GPA
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-03-03T14:55:41.423Z
Answers.Text: I also want to add that *hidden* should also have "a non-linear activation function", else it is as simple as [a linear transformation](https://ai.stackexchange.com/questions/3753/can-a-neural-network-with-linear-activation-functions-produce-a-connection-of-li/18765).
Answers.UserDisplayName: Minh-Long Luu
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-03-31T12:16:44.997Z
Answers.LastActivityDate: 2020-07-01T13:07:52.930Z
Answers.LastEditDate: 2020-07-01T13:07:52.930Z
Answers.UserDisplayName: nbro
Answers.Body: <blockquote>
  <p>Is it standard to just call deep RL to any RL algorithm that uses a neural net?</p>
  </blockquote>
  <p>Yes, it seems to have become standard practice to label RL + any NN &quot;Deep Reinforcement Learning&quot;. It is not a formalised term.</p>
  <p>The whole &quot;Deep Learning&quot; movement started this decade is as much a marketing term as a scientific one. It <em>is</em> however based on the discovery of real improvements in neural network architecture and training approaches.</p>
  <p>You may find that some (or even most) of these shallower networks will use improvements designed in the last decade or so, and also associated with deeper networks, such as Xavier initialization, ReLU activation, the Adam optimizer.</p>
  <p>As a personal opinion, I would say that, if a published experiment uses just 1 or 2 hidden layers, and does not make use of any of these recent advances, then the &quot;Deep&quot; label is almost entirely a branding exercise. There were advances with such networks much longer ago. For instance <a href="https://cling.csd.uwo.ca/cs346a/extra/tdgammon.pdf" rel="nofollow noreferrer">the TD-Gammon paper</a> is from 1995. For TD-Gammon, the authors used reinforcement learning and a NN with one hidden layer to create a Backgammon player that played better than any human player. This was well before &quot;Deep Learning&quot; was a term used to describe such networks, and the term &quot;Deep Reinforcement Learning&quot; does not appear in that paper.</p>
  <p>However, because &quot;Deep Learning&quot; is such a loose branding term, there is also an argument that all these older approaches, and pretty much all neural networks with hidden layers, should be included. <a href="https://en.wikipedia.org/wiki/Deep_learning#Definition" rel="nofollow noreferrer">Wikipedia's definition for Deep Learning</a> says:</p>
  <blockquote>
  <p>Deep learning is a class of machine learning algorithms that:</p>
  <ul>
  <li>use a cascade of multiple layers of nonlinear processing units for feature extraction and transformation. Each successive layer uses the output from the previous layer as input.</li>
  <li>learn in supervised (e.g., classification) and/or unsupervised (e.g., pattern analysis) manners.</li>
  <li>learn multiple levels of representations that correspond to different levels of abstraction; the levels form a hierarchy of concepts.</li>
  </ul>
  </blockquote>
  <p>Using that definition would include all the papers you cite.  You don't need a 50 layer Resnet architecture to qualify. And the branding exercise makes more sense under that definition, because the newly invented techniques have made such systems that much more viable and worthy of investment (of time &amp; effort as well as financially).</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-03-31T12:04:49.787Z
Answers.LastActivityDate: 2019-03-31T19:30:06.863Z
Answers.LastEditDate: 2020-06-17T09:57:20.710Z
Answers.UserDisplayName: Neil Slater
Body: <p>Often times I see the term deep reinforcement learning to refer to RL algorithms that use neural networks, regardless of whether or not the networks are deep.</p>
  <p>For example, <a href="https://arxiv.org/pdf/1707.06347.pdf" rel="nofollow noreferrer">PPO</a> is often considered a deep RL algorithm, but using a deep network is not really part of the algorithm. In fact, the example they report in the paper says that they used a network with only 2 layers.</p>
  <p>This SIGGRAPH project (<a href="https://xbpeng.github.io/projects/DeepMimic/2018_TOG_DeepMimic.pdf" rel="nofollow noreferrer">DeepMimic: Example-Guided Deep Reinforcement Learning of Physics-Based Character Skills</a>)  has the name <em>deep</em> in it and the title even says 'deep reinforcement learning', but if you read the paper, you'll see that their network uses only 2 layers.</p>
  <p>Again, the paper <a href="https://arxiv.org/pdf/1812.11103.pdf" rel="nofollow noreferrer">Learning to Walk via Deep Reinforcement Learning</a> by researchers from Google and Berkeley, contains <em>deep RL</em> in the title, but if you read the paper, you'll see they used 2 hidden layers.</p>
  <p><a href="https://www.cc.gatech.edu/%7Eaclegg3/projects/learning-dress-synthesizing.pdf" rel="nofollow noreferrer">Another SIGGRAPH project with deep RL in the title</a>. And, if you read it, surprise, 2 hidden layers.</p>
  <p>In the paper <a href="https://arxiv.org/pdf/1801.01290.pdf" rel="nofollow noreferrer">Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor</a>, if you read table 1 with the hyperparameters, they also used 2 hidden layers.</p>
  <p>Is it standard to just call deep RL to any RL algorithm that uses a neural net?</p>
  
Tags: <deep-learning><reinforcement-learning><terminology><deep-rl>
Title: Is reinforcement learning using shallow neural networks still deep reinforcement learning?
UserDisplayName: yewang
ThreadText: # Is reinforcement learning using shallow neural networks still deep reinforcement learning?
  
  (Asked by: username_0 on 2019-03-30)
  
  Often times I see the term deep reinforcement learning to refer to RL algorithms that use neural networks, regardless of whether or not the networks are deep.
  
  For example, [PPO](https://arxiv.org/pdf/1707.06347.pdf) is often considered a deep RL algorithm, but using a deep network is not really part of the algorithm. In fact, the example they report in the paper says that they used a network with only 2 layers.
  
  This SIGGRAPH project ([DeepMimic: Example-Guided Deep Reinforcement Learning of Physics-Based Character Skills](https://xbpeng.github.io/projects/DeepMimic/2018_TOG_DeepMimic.pdf)) has the name *deep* in it and the title even says 'deep reinforcement learning', but if you read the paper, you'll see that their network uses only 2 layers.
  
  Again, the paper [Learning to Walk via Deep Reinforcement Learning](https://arxiv.org/pdf/1812.11103.pdf) by researchers from Google and Berkeley, contains *deep RL* in the title, but if you read the paper, you'll see they used 2 hidden layers.
  
  [Another SIGGRAPH project with deep RL in the title](https://www.cc.gatech.edu/%7Eaclegg3/projects/learning-dress-synthesizing.pdf). And, if you read it, surprise, 2 hidden layers.
  
  In the paper [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://arxiv.org/pdf/1801.01290.pdf), if you read table 1 with the hyperparameters, they also used 2 hidden layers.
  
  Is it standard to just call deep RL to any RL algorithm that uses a neural net?
  
  **Tags:** deep-learning, reinforcement-learning, terminology, deep-rl
  
  ---
  
  ## 2 answers
  
  ### Answer by username_1 on 2019-03-31. Score: 5
  
  Even after several years of success of deep learning systems (i.e. neural networks trained with gradient descent and back-propagation), as far as I know, there is not yet a consensus on what constitutes a deep neural network. Some people could use a neural network with 2 hidden layers and call it deep (like in your case), but other people may just dedicate the adjective *deep* to refer to neural networks with 10, 100, or more hidden layers. In fact, there are some good reasons to associate the term *deep* only to neural networks that have a significant number of hidden layers (e.g. 100): for example, the exploding (or vanishing) gradient problem does not typically arise if you only have one hidden layer but can easily occur with many (e.g. 100) hidden layers.
  
  Nevertheless, a neural network with at least one hidden layer can approximate any continuous function, given enough (but finite number of) units (or neurons) in the layers. See the [universal approximation theorem](https://en.wikipedia.org/wiki/Universal_approximation_theorem). For this reason, we could start denoting any such neural network as deep, but, although this rule would exclude perceptrons (which can only approximate linear functions, and nobody would probably call them deep anyway), this rule would be a bit redundant or useless (i.e. we may just not use the adjective *deep* to start with).
  
  In your case, the rule that the authors are using seems to be the following: if it contains more hidden layers than the bare minimum (i.e. 1) to approximate any continuous function, then let's denote it as *deep*.
  
  > **Comments:**
  > **username_2**: That's a very good answer!  
  > **username_3**: I also want to add that *hidden* should also have "a non-linear activation function", else it is as simple as [a linear transformation](https://ai.stackexchange.com/questions/3753/can-a-neural-network-with-linear-activation-functions-produce-a-connection-of-li/18765).  
  
  ---
  
  ### Answer by username_4 on 2019-03-31. Score: 2
  
  > Is it standard to just call deep RL to any RL algorithm that uses a neural net?
  
  Yes, it seems to have become standard practice to label RL + any NN "Deep Reinforcement Learning". It is not a formalised term.
  
  The whole "Deep Learning" movement started this decade is as much a marketing term as a scientific one. It *is* however based on the discovery of real improvements in neural network architecture and training approaches.
  
  You may find that some (or even most) of these shallower networks will use improvements designed in the last decade or so, and also associated with deeper networks, such as Xavier initialization, ReLU activation, the Adam optimizer.
  
  As a personal opinion, I would say that, if a published experiment uses just 1 or 2 hidden layers, and does not make use of any of these recent advances, then the "Deep" label is almost entirely a branding exercise. There were advances with such networks much longer ago. For instance [the TD-Gammon paper](https://cling.csd.uwo.ca/cs346a/extra/tdgammon.pdf) is from 1995. For TD-Gammon, the authors used reinforcement learning and a NN with one hidden layer to create a Backgammon player that played better than any human player. This was well before "Deep Learning" was a term used to describe such networks, and the term "Deep Reinforcement Learning" does not appear in that paper.
  
  However, because "Deep Learning" is such a loose branding term, there is also an argument that all these older approaches, and pretty much all neural networks with hidden layers, should be included. [Wikipedia's definition for Deep Learning](https://en.wikipedia.org/wiki/Deep_learning#Definition) says:
  
  > Deep learning is a class of machine learning algorithms that:
  >
  > * use a cascade of multiple layers of nonlinear processing units for feature extraction and transformation. Each successive layer uses the output from the previous layer as input.
  > * learn in supervised (e.g., classification) and/or unsupervised (e.g., pattern analysis) manners.
  > * learn multiple levels of representations that correspond to different levels of abstraction; the levels form a hierarchy of concepts.
  
  Using that definition would include all the papers you cite. You don't need a 50 layer Resnet architecture to qualify. And the branding exercise makes more sense under that definition, because the newly invented techniques have made such systems that much more viable and worthy of investment (of time & effort as well as financially).
  
  
  ---
  
  