Question: For example, we always assumed that the data or signal error is a Gaussian distribution? why?

I have asked this question on stackoverflow, the link: <https://stackoverflow.com/questions/12616406/anyone-can-tell-me-why-we-always-use-the-gaussian-distribution-in-machine-learni>

Answer: 
I looked at the answers on SO. I don't think they are satisfactory. People often argue for the normal distribution because of the central limit theorem. That may be okay in large samples when the problem involves averages. But machine learning problems can be more complex and sample sizes are not always large enough for normal approximations to apply. Some argue for mathematical convenience. That is no justification especially when computers can easily handle added complexity and computer-intensive resampling approaches.

But I think the question should be challenged. Who says the Guassian distribution is "always" used or even just predominantly used in machine learning. Taleb claimed that statistics is dominated by the Gaussian distribution especially when applied to finance. He was very wrong about that!

In machine learning aren't kernel density classification approaches, tree classifiers and other nonparametric methods sometimes used? Aren't nearest neighbor methods used for clustering and classification? I think they are and I know statisticians use these methods very frequently.
One reason that normal distributions are often (but not always!) assumed: the nature of the distribution often leads to extremely efficient computation. For example, in generalized linear regression, the solution is technically in closed form when your distribution is Gaussian:

$\hat \beta = (X^T X)^{-1} X^T Y$

where as for other distributions, iterative algorithms must be used. Technical note: using this direct computation to find $\hat \beta$ is both inefficient and unstable.

Quite often, both the theoretical math and numerical methods required are *substaintially* easier if the distribution is a linear transformation of normal variables. Because of this, methods are frequently first developed under the assumption that the data is normal, as the problem is considerably more tractable. Later, the more difficult problem of addressing non-normality is addressed by statistical/machine learning researchers.