<p>I am newbie in MDP.I have started reading Ronald Howard Dynamic Programming and MDP book as well as Sutton and Barto An Introduction to Reinforcement Learning. To my understanding MDP is a model based (not free) process unlike SARSA and TD. The toy examples presented in both books (specially Ronald Howard's that I have run in my computer) for the value iteration have the element by element multiplication of matrices (Hadamard product) <span class="math-container">$\Pi$</span> (t.p.m) and <span class="math-container">$R$</span> (reward matrix).</p>
<p><span class="math-container">$$q = \Pi \odot R$$</span></p>
<p><span class="math-container">$q$</span> matrix might have large dimensions (columns).Is it proper to factorize this <span class="math-container">$q$</span> matrix (decompose it) ? I was thinking low rank approximations. But I have a concern: If I lose some elements of this <span class="math-container">$q$</span> matrix I will lose some paths, possible actions and policies that will not lead me to my target state. Am I correct ?</p>
<p>In addition I reading approximations of policy iteration, approximation of value iteration and so on.What do they approximate ? Do they lose some potential paths as fas I have understand ?</p>
<p>I would like some help here in order to figure out what is going on with MDP and RL in general.</p>

CC BY-SA 4.0
2023-04-17T11:59:00.427Z
Can you please put your **specific question** in the title? "Low rank approximation of MDP" is not really a question.
nbro
CC BY-SA 4.0
2023-12-26T20:20:11.070Z
Regarding "If I lose some elements of this q matrix I will lose some paths", I think your system won't reach the target, so you have to set different scenarios. By the way, what case will you loose elements?
Cloud Cho
<markov-decision-process><function-approximation><value-iteration>
Can we use Low rank approximation in Markov decision process problem?
Homer Jay Simpson
# Can we use Low rank approximation in Markov decision process problem?

(Asked by: username_0 on 2023-04-15)

I am newbie in MDP.I have started reading Ronald Howard Dynamic Programming and MDP book as well as Sutton and Barto An Introduction to Reinforcement Learning. To my understanding MDP is a model based (not free) process unlike SARSA and TD. The toy examples presented in both books (specially Ronald Howard's that I have run in my computer) for the value iteration have the element by element multiplication of matrices (Hadamard product) $\Pi$ (t.p.m) and $R$ (reward matrix).

$$q = \Pi \odot R$$

$q$ matrix might have large dimensions (columns).Is it proper to factorize this $q$ matrix (decompose it) ? I was thinking low rank approximations. But I have a concern: If I lose some elements of this $q$ matrix I will lose some paths, possible actions and policies that will not lead me to my target state. Am I correct ?

In addition I reading approximations of policy iteration, approximation of value iteration and so on.What do they approximate ? Do they lose some potential paths as fas I have understand ?

I would like some help here in order to figure out what is going on with MDP and RL in general.

**Tags:** markov-decision-process, function-approximation, value-iteration

> **Comments:**
> **username_1**: Can you please put your **specific question** in the title? "Low rank approximation of MDP" is not really a question.  
> **username_2**: Regarding "If I lose some elements of this q matrix I will lose some paths", I think your system won't reach the target, so you have to set different scenarios. By the way, what case will you loose elements?  

---

## 0 answers

