Confused on the definition of Residual standard error I am confusing with the actuarial text book. It said the residual standard error is s =RSS/(n-2), why it is not $\sqrt{\frac{RSS}{n-2}}$?
And also, what is the difference between the MSE and RSS, since $MSE= \sum_{n}^{i=1}(y_i-\hat y)^2$ and residual sum of square=$(y_i-\hat y)^2$? could I understand $\sqrt{MSE}=RSE$ ?

There is a $\frac{1}{n}$ missing in the MSE formula. Also, the summation subscript and supscript are swapped and the subscript $i$ of the prediction $\hat{y}$ is missing. It should read
$$MSE = \frac{1}{n} \sum_{i=1}^n(y_i - \hat{y}_i)^2$$
Also, $RSE = \sqrt{\frac{RSS}{dof}}$ (the square root is required, since the standard error describes a standard deviation, so no idea why it should be $RSE = \frac{RSS}{dof}$), where $RSS = \sum_{i=1}^n(y_i-\hat{y}_i)^2$ and dof is your degree of freedom (dof $
= n - 2$ for a $2$-parameter regression). Thus you have
$$MSE = \frac{1}{n}RSS = \frac{dof}{n}(RSE)^2$$
