Body: <p>I am newbie in MDP.I have started reading Ronald Howard Dynamic Programming and MDP book as well as Sutton and Barto An Introduction to Reinforcement Learning. To my understanding MDP is a model based (not free) process unlike SARSA and TD. The toy examples presented in both books (specially Ronald Howard's that I have run in my computer) for the value iteration have the element by element multiplication of matrices (Hadamard product) <span class="math-container">$\Pi$</span> (t.p.m) and <span class="math-container">$R$</span> (reward matrix).</p>
  <p><span class="math-container">$$q = \Pi \odot R$$</span></p>
  <p><span class="math-container">$q$</span> matrix might have large dimensions (columns).Is it proper to factorize this <span class="math-container">$q$</span> matrix (decompose it) ? I was thinking low rank approximations. But I have a concern: If I lose some elements of this <span class="math-container">$q$</span> matrix I will lose some paths, possible actions and policies that will not lead me to my target state. Am I correct ?</p>
  <p>In addition I reading approximations of policy iteration, approximation of value iteration and so on.What do they approximate ? Do they lose some potential paths as fas I have understand ?</p>
  <p>I would like some help here in order to figure out what is going on with MDP and RL in general.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-04-17T11:59:00.427Z
Comments.Text: Can you please put your **specific question** in the title? "Low rank approximation of MDP" is not really a question.
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-12-26T20:20:11.070Z
Comments.Text: Regarding "If I lose some elements of this q matrix I will lose some paths", I think your system won't reach the target, so you have to set different scenarios. By the way, what case will you loose elements?
Comments.UserDisplayName: Cloud Cho
Tags: <markov-decision-process><function-approximation><value-iteration>
Title: Can we use Low rank approximation in Markov decision process problem?
UserDisplayName: Homer Jay Simpson
ThreadText: # Can we use Low rank approximation in Markov decision process problem?
  
  (Asked by: username_0 on 2023-04-15)
  
  I am newbie in MDP.I have started reading Ronald Howard Dynamic Programming and MDP book as well as Sutton and Barto An Introduction to Reinforcement Learning. To my understanding MDP is a model based (not free) process unlike SARSA and TD. The toy examples presented in both books (specially Ronald Howard's that I have run in my computer) for the value iteration have the element by element multiplication of matrices (Hadamard product) $\Pi$ (t.p.m) and $R$ (reward matrix).
  
  $$q = \Pi \odot R$$
  
  $q$ matrix might have large dimensions (columns).Is it proper to factorize this $q$ matrix (decompose it) ? I was thinking low rank approximations. But I have a concern: If I lose some elements of this $q$ matrix I will lose some paths, possible actions and policies that will not lead me to my target state. Am I correct ?
  
  In addition I reading approximations of policy iteration, approximation of value iteration and so on.What do they approximate ? Do they lose some potential paths as fas I have understand ?
  
  I would like some help here in order to figure out what is going on with MDP and RL in general.
  
  **Tags:** markov-decision-process, function-approximation, value-iteration
  > **Comments:**
  > **username_1**: Can you please put your **specific question** in the title? "Low rank approximation of MDP" is not really a question.  
  > **username_2**: Regarding "If I lose some elements of this q matrix I will lose some paths", I think your system won't reach the target, so you have to set different scenarios. By the way, what case will you loose elements?  
  
  ---
  
  ## 0 answers
  
  