Question: Can you please explain when (under what circumstances) the dimensional reduction and/or feature selection techniques are not effective to mitigate the curse of dimensionality?

Answer: 
> 
> But in policy iteration also we are have to output a softmax vector related to each actions 
> 
> 
> 

This is not strictly true. A softmax vector is one possible way to represent a policy, and works for discrete action spaces. The difference between policy gradient and value function approaches here is in how you use the output. For a value function you would find the maximum output, and choose that (perhaps $\epsilon$-greedily), and it should be an estimate of the value of taking that action. For a policy function, you would use the output as probability to choose each action, and you do not know the value of taking that action.

> 
> So I don't understand how this can use to work with continuous action space ?
> 
> 
> 

With policy gradient methods, the policy can be *any* function of your parameters $\theta$ which:

* Outputs a probability distribution
* Can be differentiated with respect to $\theta$

So for instance your policy function can be 

$$\pi\_{\theta}(s) = \mathcal{N}(\mu(s,\theta), \sigma(s,\theta))$$

where $\mu$ and $\sigma$ can be functions you implement with e.g. a neural network. The output of the network is a description of the Normal distribution for the action value $a$ given a state value $s$. The policy requires you to sample from the normal distribution defined by those values (the NN doesn't do that sampling, you typically have to add that in code).

> 
> Why are policy gradient methods preferred over value function approximation in continuous action domains?
> 
> 
> 

Whilst it is still possible to estimate the *value* of a state/action pair in a continuous action space, this does not help you choose an action. Consider how you might implement an $\epsilon$-greedy policy using action value approximation: It would require performing an optimisation over the action space for each and every action choice, in order to find the estimated optimal action. This is possible, but likely to be very slow/inefficient (also there is a risk of finding local maximum).

Working directly with policies that emit probability distributions can avoid this problem, provided those distributions are easy to sample from. Hence you will often see things like policies that control parameters of the Normal distribution or similar, because it is known how to easily sample from those distributions.
In Value function methods (or Critic methods) we usually choose one of the following options to select our actions after we have estimated the relevant value function:

* **Boltzman function** with a (inverse) temperature parameter that controls how much we 'listen' the maximum of the value function to select our actions.
* **$\epsilon$-greedy:** selects the action with the maximum value function with probability (1-$\epsilon$) and a random one with probability $\epsilon$.

In Policy Gradient (or Actor Methods) we have two approaches:

1. **Stochastic Policy Gradients (SPG):** Output is a probability over actions. For your algorithm, the output would be the parameters of a pre-specified distribution (usually Gausian) as Neil described. Then you sample that distribution in a similar way you sample the Boltzman distribution.
2. **Deterministic Policy Gradients (DPG):** Output is the value of an action (e.g. speed, height etc).

From the above you can clearly see that when continuous action space is involved, PG offers much more plausible solution.

For more information I suggest you the Master thesis [PG Methods: SGD-DPG](https://riashatislam.files.wordpress.com/2016/11/riashat-islam-report.pdf) of Riashat. He worked closely with one of the co-authors of Silver's DPG paper and the thesis is very well structured with MATLAB code available.
Policy Gradients methods are much harder to understand because of the math involved (and many times the different notation used). However I would suggest you to start from [Jan Peters](http://www.scholarpedia.org/article/Policy_gradient_methods) and then get back to Silver's slides (my opinion :) ).