Body: <p>I'm attempting to implement the actor-critic algorithm on Matlab using Radial Basis Function, Local Linear Regression, and shallow Neural Network for inverted pendulum system.
  the state space and the action space are continuous.</p>
  
  <ul>
  <li>states are the angle x_1 wrapped into [-pi pi] and the angle velocity x_2 in [-8*pi 8*pi]</li>
  <li>the continuous action u, which is bound between [-3 3].</li>
  <li>reward function is quadrat rho=x'Q x+u'R u where Q=diag(1,5) and R=0.1 </li>
  <li>the desired point is upright position  [0 0]'</li>
  </ul>
  
  <hr>
  
  <p>some notes will be added</p>
  
  <ul>
  <li>the used solver is ode45.</li>
  <li>the sampling time 0.03.</li>
  <li><p>it explores random u every step, with normal distribution zero mean sigma=1</p>
  
  <p>model of the system (to save place the parameters of the model are not written)</p>
  
  <pre><code> function dy =pendulum(y,u)
  dy(1,1)=y(2);
  dy(2,1)=1/J*(M*g*l*sin(y(1))-(b+K^2/R)*y(2)+K/R*u);
  end_function
  </code></pre></li>
  </ul>
  
  <hr>
  
  <p>function to calculate RBF: the idea is to define centers and widths for N RBFs which cover the entire state space to approximate the value function and policy separately. The RBF is normalized.</p>
  
  <pre><code>function phi=RBF(x,C,B,N)         % x:state, C: centres, B: width, N: nombre of used RBfs
   Phi_vec=[];
   Phi_sum=0;
   for i=1:N                        % loop for to calculate the vector phi
       Phi_i=exp(-1/2*(x-C(i,:)')'*B^(-1)*(x-C(i,:)'));  % gaussian function
       Phi_vec=[Phi_vec;Phi_i];                    % not normalized phi vector
       Phi_sum=Phi_sum+Phi_i; % sum for normalisation
  
   end
   phi=Phi_vec/Phi_sum; % normalized phi vector
  </code></pre>
  
  <hr>
  
  <pre><code> % after tuning the learning rate for actor and critic alpha_a and alpha_c 
  
   %  every step the following updates shall be carried out: 
  
   %% generally
  
   %  Value function V=Theta_O'*RBF(x,C,B,N)
  
   %  policy pi= Theta_v'*RBF(x,C,B,N)
  
  
   % determine u(k) with exploration term
   u(k)=Theta_V'*RBF(x,C,B,N)+Delta_U(k-1)
  
   %% aplly u(k) and gain x(k+1)
  
   [t,y] = ode45(@(t,y) pendulum(y,u(k)),tspan,x(k)');
          :
   x(k+1,1)= wrapToPi(x(k+1,1)); % wrpping to pi
  
  
   % determine Temporal difference Error 
   Delta(k)=r(k)+gamma*Theta_O'*RBF(x(k),C,B,N)-Theta_O'*RBF(x(k-1),C,B,N);
  
   % eligibility trace
   z=lamda*gamma*z+RBF(x,C,B,N);
  
   % Critic update
   Theta_O=Theta_O+alpha_c*Delta(k)*z;
  
   %actor update
   Theta_V=Theta_V+alpha_a*Delta(k)*Delta_U(k- 1)*RBF(x,C,B,N);
  </code></pre>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-04-11T10:17:31.823Z
Comments.Text: Hi Khaled, welcome to AI Stack Exchange. That's a complex system you have implemented, with a lot of code and places to go wrong. In addition, the target environment could have a large impact on suitability of the solution. As it stands, there is no way to answer your question. You need to break the problem down into smaller parts, double-check how confident you are in each part, and ask questions *with far more details* about parts where you are less confident.
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-04-11T10:27:23.973Z
Comments.Text: If you want to present an overview to see if there is an obvious place to look in your implementation, then I think you need to share: (1) The nature of the environment, including states, actions and rewards. (2) What fields are in your raw data. (3) How you have applied RBF to that data (especially how you have normalised or scaled it). (4) A learning graph for actor-critic. Use [edit] to add those details to the current question
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-04-11T15:31:20.070Z
Comments.Text: thank you for considering my post and advice me to improve it
Comments.UserDisplayName: Khaled
Tags: <neural-networks><reinforcement-learning><linear-regression><actor-critic-methods>
Title: Actor-critic algorithm using gaussian Radial Basis Function, Local Linear Regression and shallow Neural Network
UserDisplayName: Khaled
ThreadText: # Actor-critic algorithm using gaussian Radial Basis Function, Local Linear Regression and shallow Neural Network
  
  (Asked by: username_0 on 2019-04-11)
  
  I'm attempting to implement the actor-critic algorithm on Matlab using Radial Basis Function, Local Linear Regression, and shallow Neural Network for inverted pendulum system.
  the state space and the action space are continuous.
  
  * states are the angle x\_1 wrapped into [-pi pi] and the angle velocity x\_2 in [-8\*pi 8\*pi]
  * the continuous action u, which is bound between [-3 3].
  * reward function is quadrat rho=x'Q x+u'R u where Q=diag(1,5) and R=0.1
  * the desired point is upright position [0 0]'
  
  ---
  
  some notes will be added
  
  * the used solver is ode45.
  * the sampling time 0.03.
  * it explores random u every step, with normal distribution zero mean sigma=1
  
    model of the system (to save place the parameters of the model are not written)
  
    ```
     function dy =pendulum(y,u)
    dy(1,1)=y(2);
    dy(2,1)=1/J*(M*g*l*sin(y(1))-(b+K^2/R)*y(2)+K/R*u);
    end_function
  
    ```
  
  ---
  
  function to calculate RBF: the idea is to define centers and widths for N RBFs which cover the entire state space to approximate the value function and policy separately. The RBF is normalized.
  
  ```
  function phi=RBF(x,C,B,N)         % x:state, C: centres, B: width, N: nombre of used RBfs
   Phi_vec=[];
   Phi_sum=0;
   for i=1:N                        % loop for to calculate the vector phi
       Phi_i=exp(-1/2*(x-C(i,:)')'*B^(-1)*(x-C(i,:)'));  % gaussian function
       Phi_vec=[Phi_vec;Phi_i];                    % not normalized phi vector
       Phi_sum=Phi_sum+Phi_i; % sum for normalisation
  
   end
   phi=Phi_vec/Phi_sum; % normalized phi vector
  
  ```
  
  ---
  
  ```
   % after tuning the learning rate for actor and critic alpha_a and alpha_c 
  
   %  every step the following updates shall be carried out: 
  
   %% generally
  
   %  Value function V=Theta_O'*RBF(x,C,B,N)
  
   %  policy pi= Theta_v'*RBF(x,C,B,N)
  
   % determine u(k) with exploration term
   u(k)=Theta_V'*RBF(x,C,B,N)+Delta_U(k-1)
  
   %% aplly u(k) and gain x(k+1)
  
   [t,y] = ode45(@(t,y) pendulum(y,u(k)),tspan,x(k)');
          :
   x(k+1,1)= wrapToPi(x(k+1,1)); % wrpping to pi
  
   % determine Temporal difference Error 
   Delta(k)=r(k)+gamma*Theta_O'*RBF(x(k),C,B,N)-Theta_O'*RBF(x(k-1),C,B,N);
  
   % eligibility trace
   z=lamda*gamma*z+RBF(x,C,B,N);
  
   % Critic update
   Theta_O=Theta_O+alpha_c*Delta(k)*z;
  
   %actor update
   Theta_V=Theta_V+alpha_a*Delta(k)*Delta_U(k- 1)*RBF(x,C,B,N);
  
  ```
  
  **Tags:** neural-networks, reinforcement-learning, linear-regression, actor-critic-methods
  
  > **Comments:**
  > **username_1**: Hi username_0, welcome to AI Stack Exchange. That's a complex system you have implemented, with a lot of code and places to go wrong. In addition, the target environment could have a large impact on suitability of the solution. As it stands, there is no way to answer your question. You need to break the problem down into smaller parts, double-check how confident you are in each part, and ask questions *with far more details* about parts where you are less confident.  
  > **username_1**: If you want to present an overview to see if there is an obvious place to look in your implementation, then I think you need to share: (1) The nature of the environment, including states, actions and rewards. (2) What fields are in your raw data. (3) How you have applied RBF to that data (especially how you have normalised or scaled it). (4) A learning graph for actor-critic. Use [edit] to add those details to the current question  
  > **username_0**: thank you for considering my post and advice me to improve it  
  
  ---
  
  ## 0 answers
  
  