Answers.Body: <ol>
  <li>It's not possible, as in the chain illustrated there are no transitions between A and C, C and D, and D and F. Only sequences where transitions exist are possible. The choice of transitions is arbitrary; it depends on what you want to model with it.</li>
  </ol>
  
  <p>As DuttaA says in his comment, you can imagine that all nodes are linked with all other nodes, but those links have a transition probability of 0.0; so the probability of observing the sequence ACDF is actually 0.0 as well. In order to make the diagram more easy to understand, zero-probability transitions are generally not shown.</p>
  
  <ol start="2">
  <li>There is no prescribed method of populating the transition probabilities. You can define the probabilities manually, randomly, or however you like. You could derive them from observable training data.</li>
  </ol>
  
  <p>Re updating: this depends on the application. If you are modelling a process with known probabilities, you would not update the probabilities; if you are modelling a dynamic process which changes over time, then this is something you might want to consider. However, from my own experience (HMMs in speech recognition), once they have been assigned, they are generally kept as they are.</p>
  
  <p>A <a href="https://en.wikipedia.org/wiki/Markov_chain" rel="nofollow noreferrer">Markov chain</a> is really a fairly basic model; it gets more complex with a <a href="https://en.wikipedia.org/wiki/Hidden_Markov_model" rel="nofollow noreferrer">Hidden Markov Model</a>, where you would generally use a learning algorithm to assign the transition and emission probabilities.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-02-29T21:15:12.003Z
Answers.Text: Yes, DuttaA, thanks. So I guess the transitions are dictated in the Probability Transition Matrix? If so, then I presume the sequences are set as Oliver Mason says in point 2. So, if they’re set randomly, how are the probabilities updated to their ‘real’ probabilities (or does that not happen?).
Answers.UserDisplayName: mason7663
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-02-29T21:21:27.603Z
Answers.Text: @DuttaA Thanks for your comments; I have extended my answer.
Answers.UserDisplayName: Oliver Mason
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-02-29T21:57:09.933Z
Answers.Text: Thanks for updating, Oliver. If, for example, the environment is a board game and its states are the various position the game pieces may be in, how would the PTM be initialised? How would it be (if it is?) updated?
Answers.UserDisplayName: mason7663
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-02-29T22:12:17.017Z
Answers.Text: It's probably best to put that in a new question; and give a concrete example of a game.
Answers.UserDisplayName: Oliver Mason
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-02-29T17:38:30.027Z
Answers.LastActivityDate: 2020-02-29T21:18:06.833Z
Answers.LastEditDate: 2020-02-29T21:18:06.833Z
Answers.UserDisplayName: Oliver Mason
Body: <p>I'm watching the David Silver series on YT which has raised a couple of questions: </p>
  
  <p><strong>In the Markov process (or chain), how are the directions to each successive state defined?</strong> For example, how are the arrow directions defined for the MP below? What's stopping our sample episodes from choosing <code>A -&gt; C -&gt; D -&gt; F</code>?</p>
  
  <p><a href="https://i.sstatic.net/PbKgcm.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/PbKgcm.png" alt="enter image description here"></a></p>
  
  <p>Also, <strong>how is the probability transition matrix populated?</strong> From David's example, the probabilities seem to have already been set. For example:</p>
  
  <p><a href="https://i.sstatic.net/QJoawm.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/QJoawm.png" alt="enter image description here"></a></p>
  
Tags: <machine-learning><reinforcement-learning><markov-chain>
Title: In the Markov chain, how are the directions to each successive state defined?
UserDisplayName: mason7663
ThreadText: # In the Markov chain, how are the directions to each successive state defined?
  
  (Asked by: username_0 on 2020-02-29)
  
  I'm watching the David Silver series on YT which has raised a couple of questions:
  
  **In the Markov process (or chain), how are the directions to each successive state defined?** For example, how are the arrow directions defined for the MP below? What's stopping our sample episodes from choosing `A -> C -> D -> F`?
  
  [![enter image description here](https://i.sstatic.net/PbKgcm.png)](https://i.sstatic.net/PbKgcm.png)
  
  Also, **how is the probability transition matrix populated?** From David's example, the probabilities seem to have already been set. For example:
  
  [![enter image description here](https://i.sstatic.net/QJoawm.png)](https://i.sstatic.net/QJoawm.png)
  
  **Tags:** machine-learning, reinforcement-learning, markov-chain
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2020-02-29. Score: 4 | Accepted answer
  
  1. It's not possible, as in the chain illustrated there are no transitions between A and C, C and D, and D and F. Only sequences where transitions exist are possible. The choice of transitions is arbitrary; it depends on what you want to model with it.
  
  As DuttaA says in his comment, you can imagine that all nodes are linked with all other nodes, but those links have a transition probability of 0.0; so the probability of observing the sequence ACDF is actually 0.0 as well. In order to make the diagram more easy to understand, zero-probability transitions are generally not shown.
  
  2. There is no prescribed method of populating the transition probabilities. You can define the probabilities manually, randomly, or however you like. You could derive them from observable training data.
  
  Re updating: this depends on the application. If you are modelling a process with known probabilities, you would not update the probabilities; if you are modelling a dynamic process which changes over time, then this is something you might want to consider. However, from my own experience (HMMs in speech recognition), once they have been assigned, they are generally kept as they are.
  
  A [Markov chain](https://en.wikipedia.org/wiki/Markov_chain) is really a fairly basic model; it gets more complex with a [Hidden Markov Model](https://en.wikipedia.org/wiki/Hidden_Markov_model), where you would generally use a learning algorithm to assign the transition and emission probabilities.
  
  > **Comments:**
  > **username_0**: Yes, DuttaA, thanks. So I guess the transitions are dictated in the Probability Transition Matrix? If so, then I presume the sequences are set as username_1 username_1 says in point 2. So, if they’re set randomly, how are the probabilities updated to their ‘real’ probabilities (or does that not happen?).  
  > **username_1**: @DuttaA Thanks for your comments; I have extended my answer.  
  > **username_0**: Thanks for updating, username_1. If, for example, the environment is a board game and its states are the various position the game pieces may be in, how would the PTM be initialised? How would it be (if it is?) updated?  
  > **username_1**: It's probably best to put that in a new question; and give a concrete example of a game.  
  
  ---
  
  