Answers.Body: <p>Many real-world problems have an infinite number of states, but, in practice, digital computers cannot represent an infinite number of states or numbers anyway, so you will always need to discretize the problems (or you will do that implicitly).</p>
  <p>Now, think of an image, where each pixel can be a number between 0 and 1.0. If we used real numbers, there could be an infinite number of images, even if we restrict our images to finite shapes. So, in theory, we could have an infinite number of states. In practice, when we use real computers, we will never have an infinite number of states.</p>
  <p>Having said this, yes, RL has been applied to problems where, in theory, the number of states could be infinite. For example, <a href="https://arxiv.org/pdf/1312.5602.pdf" rel="nofollow noreferrer">DQN</a> was trained and used to play video games given a sequence of images (observations) as states. You could argue that pixels can also just have integer values between 0 and 255. In that case, you can think of problems like the pendulum or the mountain car.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-02-13T12:45:13.820Z
Answers.LastActivityDate: 2024-02-13T12:45:13.820Z
Answers.UserDisplayName: nbro
Body: <p>I was learning fundamentals of reinforcement learning from various sources like <a href="https://www.coursera.org/learn/fundamentals-of-reinforcement-learning/home/week/1" rel="nofollow noreferrer"><code>coursera</code></a>, <a href="https://www.udacity.com/course/deep-reinforcement-learning-nanodegree--nd893" rel="nofollow noreferrer"><code>Udacity</code></a>, <a href="https://web.stanford.edu/class/psych209/Readings/SuttonBartoIPRLBook2ndEd.pdf" rel="nofollow noreferrer"><code>Reinforcement Learning: An Introduction Book by Andrew Barto and Richard S. Sutton</code></a>.</p>
  <p>All of them consider Finite MDPs as reinforcement learning problem and try to solve that.</p>
  <p>In real life, is there any example of an infinite MDP (MDP with infinite number of states) and if yes, can it be described and solved as a reinforcement learning problem?</p>
  <p>Can anyone suggest some sources to read more about this problem statement?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-02-12T07:33:26.197Z
Comments.Text: Could you define what you mean by *infinite* MDP? There are a few traits of finite MDPs that can be changed such that they are not finite/discrete and they can be solved using different approaches. For instance, there is "inifnite horizon MDP", i.e. MDP which never terminates, and you want to consider all future results. Are you asking about that?
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-02-12T08:28:11.797Z
Comments.Text: Yes, exactly. I was talking about MDPs which never terminate.
Comments.UserDisplayName: Bhavesh Achhada
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-02-12T11:02:20.103Z
Comments.Text: **Finite MDPs are not defined as MDPs with infinite-horizon**. Finite MDPs are defined as MDPs with a finite set of states and actions (and maybe rewards too). This is the definition adopted by Barto and Sutton and many other sources I've come across. Also, the same book covers infinite-horizon MDPs, assuming finite MDPs. So, please, edit your post to clarify what you're really asking here. I'd recommend that, before doing that, you read the sections of this book that define finite-MDPs and infinite-horizon settings.
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-02-12T15:22:08.537Z
Comments.Text: I have updated the question with more clarification. I will definitely read upon the concept of horizon.
Comments.UserDisplayName: Bhavesh Achhada
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-02-12T15:23:44.813Z
Comments.Text: Not really an answer, but it will lead you to it, - Yes, look at continuous state and action spaces, they are both infinite in size and a common RL problem.
Comments.UserDisplayName: Vaas
Tags: <reinforcement-learning><markov-decision-process>
Title: Is there any example of a Markov Decision Process (MDP) with infinite number of states?
UserDisplayName: Bhavesh Achhada
ThreadText: # Is there any example of a Markov Decision Process (MDP) with infinite number of states?
  
  (Asked by: username_0 on 2024-02-12)
  
  I was learning fundamentals of reinforcement learning from various sources like [`coursera`](https://www.coursera.org/learn/fundamentals-of-reinforcement-learning/home/week/1), [`Udacity`](https://www.udacity.com/course/deep-reinforcement-learning-nanodegree--nd893), [`Reinforcement Learning: An Introduction Book by Andrew Barto and Richard S. Sutton`](https://web.stanford.edu/class/psych209/Readings/SuttonBartoIPRLBook2ndEd.pdf).
  
  All of them consider Finite MDPs as reinforcement learning problem and try to solve that.
  
  In real life, is there any example of an infinite MDP (MDP with infinite number of states) and if yes, can it be described and solved as a reinforcement learning problem?
  
  Can anyone suggest some sources to read more about this problem statement?
  
  **Tags:** reinforcement-learning, markov-decision-process
  
  > **Comments:**
  > **username_1**: Could you define what you mean by *infinite* MDP? There are a few traits of finite MDPs that can be changed such that they are not finite/discrete and they can be solved using different approaches. For instance, there is "inifnite horizon MDP", i.e. MDP which never terminates, and you want to consider all future results. Are you asking about that?  
  > **username_0**: Yes, exactly. I was talking about MDPs which never terminate.  
  > **username_2**: **Finite MDPs are not defined as MDPs with infinite-horizon**. Finite MDPs are defined as MDPs with a finite set of states and actions (and maybe rewards too). This is the definition adopted by Barto and Sutton and many other sources I've come across. Also, the same book covers infinite-horizon MDPs, assuming finite MDPs. So, please, edit your post to clarify what you're really asking here. I'd recommend that, before doing that, you read the sections of this book that define finite-MDPs and infinite-horizon settings.  
  > **username_0**: I have updated the question with more clarification. I will definitely read upon the concept of horizon.  
  > **username_3**: Not really an answer, but it will lead you to it, - Yes, look at continuous state and action spaces, they are both infinite in size and a common RL problem.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2024-02-13. Score: 2 | Accepted answer
  
  Many real-world problems have an infinite number of states, but, in practice, digital computers cannot represent an infinite number of states or numbers anyway, so you will always need to discretize the problems (or you will do that implicitly).
  
  Now, think of an image, where each pixel can be a number between 0 and 1.0. If we used real numbers, there could be an infinite number of images, even if we restrict our images to finite shapes. So, in theory, we could have an infinite number of states. In practice, when we use real computers, we will never have an infinite number of states.
  
  Having said this, yes, RL has been applied to problems where, in theory, the number of states could be infinite. For example, [DQN](https://arxiv.org/pdf/1312.5602.pdf) was trained and used to play video games given a sequence of images (observations) as states. You could argue that pixels can also just have integer values between 0 and 255. In that case, you can think of problems like the pendulum or the mountain car.
  
  
  ---
  
  