Which weights are the weights of parameters of Linear Regression? 
I have just started studying Linear Regression using Neural Network and I believe this is a naive question.
I have a neural network for a Linear Regression Model with 3 independent variables. I have 1 hidden layer with 3 neurons with linear activations for each neuron. The last neuron is just another linear activation function that sums the three outputs from the hidden layer and compare the predicted value with the actual value.
I am wondering which weights are the final weights between the weights before the hidden layer and the weights after the hidden layer that I can use for the parameters of the Linear Regression Model. Hope to hear some explanations.

Since your network has linear activation functions it can be written as a linear regression (it couldn't be if it used other activation functions).  If write $w_{1i}$ for the weight of $x_i$ in neuron $h_1$ and $v_i$ for the weight of the output of $h_i$ in the final output, and $\beta_i$ for the coefficient of $x_i$ in a linear regression, then
$$\beta_1=\sum_{j=1}^3 v_jw_{j1}$$
$$\beta_2=\sum_{j=1}^3 v_jw_{j2}$$
$$\beta_3=\sum_{j=1}^3 v_jw_{j3}$$
The estimates will be the same as the OLS estimates if you use in-sample squared error as your loss function and if you actually find the globally optimal $w$ and $v$. I don't know whether back-propagation guarantees finding the global optimum in this very simple case; it doesn't in more complicated cases.
