Generalized Linear Model: Why do we insist on modeling the mean? A standard description of GLM is said to have three elements:

*

*The linear predictor $\eta=X \beta$

*The link function $g(.)$ that relates the linear predictor to the mean of the outcome variable $\mu=g^{-1}(\eta)=g^{-1}(X\beta)$

*A distribution of the outcome $y$ with its mean $\mathbb E(y|X)=\mu$
See Wikipedia and Gelman et al.'s Bayesian Data Analysis 3rd edition for some examples.
My question is why is the mean of the distribution so special. Instead of modeling $\mathbb E(Y|X)=g^{-1}(X \beta)$, why can't we model other parameters of the distribution? That is $\text{(other parameters)}=g^{-1}(X \beta)$?
For example, the parameters of the Weibull distribution do not correspond to its mean, but I see models that look like this:
$$
\begin{aligned}
y & \sim Wei(v, \lambda)  \\
\ln(\lambda) &= X \beta  
\end{aligned}
$$
As another example, the mixture density network models the parameters of a mixture of Gaussians.
Nevertheless, every book I read about GLM says that we model the mean $\mathbb E(Y|X)=g^{-1}(X \beta)$. It makes me wonder that I am missing something important.

Generalized linear models are generalization of linear regression model. Linear regression minimizes squared error for fitting the regression line. Minimizing squared error is equivalent to maximizing Gaussian likelihood. GLMs generalize this idea to other distributions, so we are estimating conditional means of other distributions.
So it’s a little bit like asking “what’s the fuss with bikes having two wheels”, well they’re kind of vehicles that do. As others noticed in the comments, there are many other models that not necessarily focus on means. GLMs are just a family of models that do.
