Answers.Body: <p>If your goal is to create a controller for the mountain car problem, and you have access to the model, then RL <em>probably</em> offers no advantage over your code. I am saying probably, because I am taking you at your word that the code performs well over multiple tests, and it doesn't matter too much if it does not because there are many equivalent solutions based on analysis of the original problem.</p>
  <p>This is the difference:</p>
  <ul>
  <li><p>RL techniques <em>find</em> solutions to control problems. Model-free RL techniques can do so without access to the environment model.</p>
  </li>
  <li><p>The classic control code <em>is</em> a solution to a given control problem, found by the code author through analysis of the problem.</p>
  </li>
  </ul>
  <p>The same RL agent that could solve mountain car, could solve similar environments with same state and action space, e.g. an environment similar to mountain car but with multiple hills and valleys, or with alterations to the physics model. The same classic control code would fail and need to be re-written for the new environment.</p>
  <p>The mountain car problem is interesting in control theory because it introduces a level of abstraction - the simplest feedback-based control algorithms will fail because moving directly towards the goal does not work. However, it is still a toy problem. The solutions are well understood, and no-one needs to solve it again. Solving it with RL is not necessary, it is a demonstration of learning something through trial and error.</p>
  <p>As control problems become more complex, with multiple levels of goals to solve, then classic control approaches become more unweidly. For example, a walking robot has many more variables to manage, and walking systems are more likely to benefit from automated search for the best controllers as opposed to analysis and classic control at all levels.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-07-25T08:55:53.743Z
Answers.LastActivityDate: 2021-07-25T11:50:34.277Z
Answers.LastEditDate: 2021-07-25T11:50:34.277Z
Answers.UserDisplayName: Neil Slater
Body: <p>What is the advantage of RL compared with the following simple classic algorithm for the <code>MountainCarEnv</code>? Considering that it takes a long time to train the agent just to achieve this simple task?</p>
  <pre><code>import gym
  
  envName = 'MountainCar-v0'
  env = gym.make(envName)
  
  x, v = state = env.reset()
  done = False
  maxPotential = False
  steps = 0
  
  def computeAction(state):
      global x, v, maxPotential, steps
      xNew, vNew = state
      action = 1
      if xNew &lt; -1.1:
          maxPotential = True
      if not maxPotential:
          if xNew &lt; x: action = 0
          else: action = 2
      else:
          action = 2
      x, v = xNew, vNew
      steps += 1
      return action
  
  while not done:
      state, reward, done, info = env.step(computeAction(state))
      env.render()
  
  print('steps', steps)
  </code></pre>
  <ul>
  <li>result: around 100 steps</li>
  </ul>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-07-25T04:07:45.727Z
Comments.Text: Hello, welcome to Artificial Intelligence Stack Exchange. If possible, please mention the technique used by the code you posted. Is it RL? If yes, then between what techniques of RL you are asking for comparison.
Comments.UserDisplayName: hanugm
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-07-25T04:56:32.747Z
Comments.Text: @hanugm, nope, if you just take a quick look at the code, this is a simple policy based on physical intuition, in this case I am comparing RL with classical control method, i.e. the code above
Comments.UserDisplayName: em1971
Tags: <reinforcement-learning><comparison>
Title: What is the advantage of RL compared with my simple classic algorithm for the MountainCarEnv?
UserDisplayName: em1971
ThreadText: # What is the advantage of RL compared with my simple classic algorithm for the MountainCarEnv?
  
  (Asked by: username_0 on 2021-07-25)
  
  What is the advantage of RL compared with the following simple classic algorithm for the MountainCarEnv? Considering that it takes a long time to train the agent just to achieve this simple task?
  import gym
  
  envName = 'MountainCar-v0'
  env = gym.make(envName)
  
  x, v = state = env.reset()
  done = False
  maxPotential = False
  steps = 0
  
  def computeAction(state):
      global x, v, maxPotential, steps
      xNew, vNew = state
      action = 1
      if xNew < -1.1:
          maxPotential = True
      if not maxPotential:
          if xNew < x: action = 0
          else: action = 2
      else:
          action = 2
      x, v = xNew, vNew
      steps += 1
      return action
  
  while not done:
      state, reward, done, info = env.step(computeAction(state))
      env.render()
  
  print('steps', steps)
  
  
  result: around 100 steps
  
  **Tags:** reinforcement-learning, comparison
  > **Comments:**
  > **username_1**: Hello, welcome to Artificial Intelligence Stack Exchange. If possible, please mention the technique used by the code you posted. Is it RL? If yes, then between what techniques of RL you are asking for comparison.  
  > **username_0**: @hanugm, nope, if you just take a quick look at the code, this is a simple policy based on physical intuition, in this case I am comparing RL with classical control method, i.e. the code above  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2021-07-25. Score: 1
  
  If your goal is to create a controller for the mountain car problem, and you have access to the model, then RL *probably* offers no advantage over your code. I am saying probably, because I am taking you at your word that the code performs well over multiple tests, and it doesn't matter too much if it does not because there are many equivalent solutions based on analysis of the original problem.
  
  This is the difference:
  
  * RL techniques *find* solutions to control problems. Model-free RL techniques can do so without access to the environment model.
  * The classic control code *is* a solution to a given control problem, found by the code author through analysis of the problem.
  
  The same RL agent that could solve mountain car, could solve similar environments with same state and action space, e.g. an environment similar to mountain car but with multiple hills and valleys, or with alterations to the physics model. The same classic control code would fail and need to be re-written for the new environment.
  
  The mountain car problem is interesting in control theory because it introduces a level of abstraction - the simplest feedback-based control algorithms will fail because moving directly towards the goal does not work. However, it is still a toy problem. The solutions are well understood, and no-one needs to solve it again. Solving it with RL is not necessary, it is a demonstration of learning something through trial and error.
  
  As control problems become more complex, with multiple levels of goals to solve, then classic control approaches become more unweidly. For example, a walking robot has many more variables to manage, and walking systems are more likely to benefit from automated search for the best controllers as opposed to analysis and classic control at all levels.
  
  
  ---
  
  