Relationship between system of equations and statistical models I am studying statistics and I am starting to see some parallels between statistical models and systems of equations I used to solve in high school.
For example, suppose that on one hand we have a linear system of equations x + 3y = 4, 2x + 4 =2 and on the other hand we have 10 observations of two normal variables W and Z that we assume come from the model W = b0 + b1Z + error (standard linear regression). In both cases we have:

*

*In both cases there are unknowns that we are trying to find (x, y, b0, b1, var(error))

*In both cases there ares some constraints for the unknowns (W, Z ~ Normal, x,y belong to the real or complex numbers)

*Once the system is formulated there exist deterministic algorithms to solve it.

My question is the following: is there a general framework that encompasses both problems?
For example:

*

*Is a system of equations a particular case of a statistical model, if we consider degenerated distributions (e.g., if we interpret X=3 as P(X=3)=1)?

*Is there a parallel between a system of equations having infinite solutions and a statistical model to be identifiable? What about a system of equations having no solution, what would be the equivalent in a statistical model?

*I have the feeling that the concept of degrees of freedom is important in this parallel, but I don't know exactly how.




*

*Is a system of equations a particular case of a statistical model, ...


The estimation of a statistical model based on a linear system of equations is called linear regression.



*Is there a parallel between a system of equations having infinite solutions and a statistical model to be identifiable? What about a system of equations having no solution, what would be the equivalent in a statistical model?


You can regard the number of parameters $p$ and the number of measurements $n$.
(these $n$ and $p$ can be seen as parallel to the number of equations $n$ and number of unknowns $p$)
Often we have more measurements than parameters. For the equivalent linear equation there might be no solution. But in the case of least squares regression this is no problem. We search for the solution that fits the best.
If there are an infinite number of solutions, $p\geq n$, then additional constraints are used. See for instance ridge regression where the solution with the smallest length of the parameter vector is chosen.



*I have the feeling that the concept of degrees of freedom is important in this parallel, but I don't know exactly how.


I don't know how to answer this.
