Question: For example, we always assumed that the data or signal error is a Gaussian distribution? why?

I have asked this question on stackoverflow, the link: <https://stackoverflow.com/questions/12616406/anyone-can-tell-me-why-we-always-use-the-gaussian-distribution-in-machine-learni>

Answer: 
One reason that normal distributions are often (but not always!) assumed: the nature of the distribution often leads to extremely efficient computation. For example, in generalized linear regression, the solution is technically in closed form when your distribution is Gaussian:

$\hat \beta = (X^T X)^{-1} X^T Y$

where as for other distributions, iterative algorithms must be used. Technical note: using this direct computation to find $\hat \beta$ is both inefficient and unstable.

Quite often, both the theoretical math and numerical methods required are *substaintially* easier if the distribution is a linear transformation of normal variables. Because of this, methods are frequently first developed under the assumption that the data is normal, as the problem is considerably more tractable. Later, the more difficult problem of addressing non-normality is addressed by statistical/machine learning researchers.
i'm currently studying machine learning and the same question popped to my mind. What I think the reason should be is that in every machine learning problem we assume we have abundant observational data available and whenever data tends to infinity it gets normally distributed around its mean and thats what Normal distribution(Gaussian Distribution) says. Although its not necessary that Gaussian DIstribution will always be a perfect fit to any data that tends to infinity like take a case when your data is always positive then you can easily see if you try to fit gaussian dist to it it will give some weight to negative values of x also(although negligible but still some weight is given) so in such case distribution like Zeta is more suited.