the probability of generating a particular vector v, is just the sum of all the possible hidden states of the probability of generating those hidden state times the probability of generating v, given that you've already generated that hidden state.