Body: <p>I want to customize the 'Pendulum-v0' environment such that the action (the torque) from previous time step as well as from the current timestep serve as the inputs in the Env.step() function.</p>
  
  <p>My problem statement is that I want to generate torque from the controller which has a white Gaussian noise of magnitude 1 and then filter it with the torque generated in the previous timestep as follows:</p>
  
  <p>tor_ = tor_c + a*WGN ;</p>
  
  <p>tor(t) = lambda*tor_ + (1-lambda)*tor(t-1) ;</p>
  
  <p><a href="https://github.com/openai/gym/blob/master/gym/envs/classic_control/pendulum.py#L37" rel="nofollow noreferrer">https://github.com/openai/gym/blob/master/gym/envs/classic_control/pendulum.py#L37</a></p>
  
  <p>We can see that the 'u' is an array of some numbers as the input which is afterward clipped from -max_torque to max_torque and then only the first element is taken as the torque value to calculate the states and the reward function for the given time-step.</p>
  
  <p>My question is what does the value of the other elements signify? Are they
  torque values from the previous time steps or is it that the length of the 'u' array is just 1 and its value is restricted between -max_torque to max_torque?</p>
  
  <p>In conclusion, I just wanna access the action (the torque value) from the previous time-step. Is it possible ? If yes, how?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-02-05T12:36:12.713Z
Comments.Text: there are no other elements, u is an iterable with a single element (action that we want to take) that we send to step function. They use np.clip function to clip the number in given range and the np.clip function returns np.array (with only 1 element). Then they retrieve the float from it with [0].
Comments.UserDisplayName: Brale
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-02-05T17:54:12.593Z
Comments.Text: So how can I access the action from the previous time step for low pass filtering of the torque? Aren't actions from every time steps saved somewhere in the action space ?
Comments.UserDisplayName: Shritej Chavan
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-02-05T19:23:16.133Z
Comments.Text: You provide actions to the environment from your code, simply save the action from the previous loop iteration in your code and use it for the update.
Comments.UserDisplayName: Brale
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-02-07T05:52:46.750Z
Comments.Text: Also, I wanna model it such that the pendulum is always in -pi/8 to pi/8 (with the vertical) and if it gets beyond that, the environment resets. So I will define the observation space such that there is bound on theta rather than using cosine or sine function as used.in 'Pendulum-v0'. Is it okay do it this way? (Sorry if this is a noob question)
Comments.UserDisplayName: Shritej Chavan
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-02-07T22:07:45.967Z
Comments.Text: I don't think they actually have termination condition, environment just keeps on going, so the only current termination condition would be to limit the number of timesteps. Regarding limiting the angle for reset condition, I think that would be fine, except that in this case pendulums starting position is at random angle so with your condition environment could immediately reset, assuming that you meant pi/8 offset from pi/2 angle ?
Comments.UserDisplayName: Brale
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-02-09T04:37:59.897Z
Comments.Text: In my case there is no problem in using theta and theta dot as the observation space for the environment, right? And what do u mean by termination condition? And speaking of starting position, the pendulum will be initialized at random angle between -5 to +5 degrees with the pi/2 angle.
Comments.UserDisplayName: Shritej Chavan
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-02-10T09:03:44.420Z
Comments.Text: I think using theta and theta dot would be fine, best way to find out is to try out. I think that you might be confusing pendulum-v0 with cartpole-v0 environment. Pendulum starts at bottom and tries to swing up and stay at top, while cartpole starts at top and tries to stay there without falling down.
Comments.UserDisplayName: Brale
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-02-10T21:40:26.533Z
Comments.Text: But I want it the way cartpole is designed to start not the way pendulum is..
Comments.UserDisplayName: Shritej Chavan
Tags: <reinforcement-learning>
Title: Input for the Env.step() in the 'Pendulum-v0' environment
UserDisplayName: Shritej Chavan
ThreadText: # Input for the Env.step() in the 'Pendulum-v0' environment
  
  (Asked by: username_0 on 2019-02-05)
  
  I want to customize the 'Pendulum-v0' environment such that the action (the torque) from previous time step as well as from the current timestep serve as the inputs in the Env.step() function.
  
  My problem statement is that I want to generate torque from the controller which has a white Gaussian noise of magnitude 1 and then filter it with the torque generated in the previous timestep as follows:
  
  tor\_ = tor\_c + a\*WGN ;
  
  tor(t) = lambda\*tor\_ + (1-lambda)\*tor(t-1) ;
  
  <https://github.com/openai/gym/blob/master/gym/envs/classic_control/pendulum.py#L37>
  
  We can see that the 'u' is an array of some numbers as the input which is afterward clipped from -max\_torque to max\_torque and then only the first element is taken as the torque value to calculate the states and the reward function for the given time-step.
  
  My question is what does the value of the other elements signify? Are they
  torque values from the previous time steps or is it that the length of the 'u' array is just 1 and its value is restricted between -max\_torque to max\_torque?
  
  In conclusion, I just wanna access the action (the torque value) from the previous time-step. Is it possible ? If yes, how?
  
  **Tags:** reinforcement-learning
  > **Comments:**
  > **username_1**: there are no other elements, u is an iterable with a single element (action that we want to take) that we send to step function. They use np.clip function to clip the number in given range and the np.clip function returns np.array (with only 1 element). Then they retrieve the float from it with [0].  
  > **username_0**: So how can I access the action from the previous time step for low pass filtering of the torque? Aren't actions from every time steps saved somewhere in the action space ?  
  > **username_1**: You provide actions to the environment from your code, simply save the action from the previous loop iteration in your code and use it for the update.  
  > **username_0**: Also, I wanna model it such that the pendulum is always in -pi/8 to pi/8 (with the vertical) and if it gets beyond that, the environment resets. So I will define the observation space such that there is bound on theta rather than using cosine or sine function as used.in 'Pendulum-v0'. Is it okay do it this way? (Sorry if this is a noob question)  
  > **username_1**: I don't think they actually have termination condition, environment just keeps on going, so the only current termination condition would be to limit the number of timesteps. Regarding limiting the angle for reset condition, I think that would be fine, except that in this case pendulums starting position is at random angle so with your condition environment could immediately reset, assuming that you meant pi/8 offset from pi/2 angle ?  
  > **username_0**: In my case there is no problem in using theta and theta dot as the observation space for the environment, right? And what do u mean by termination condition? And speaking of starting position, the pendulum will be initialized at random angle between -5 to +5 degrees with the pi/2 angle.  
  > **username_1**: I think using theta and theta dot would be fine, best way to find out is to try out. I think that you might be confusing pendulum-v0 with cartpole-v0 environment. Pendulum starts at bottom and tries to swing up and stay at top, while cartpole starts at top and tries to stay there without falling down.  
  > **username_0**: But I want it the way cartpole is designed to start not the way pendulum is..  
  
  ---
  
  ## 0 answers
  
  