Is it necessary to assume normal errors and independence in a linear regression model? I am reading Casella Berger and in the book (pg. 545), they mention that it is possible to construct a linear regression model by only assuming that:
$$
E(Y_i) = \alpha+\beta X_i \qquad \text{with} \qquad Var(Y_i) = \sigma^2
$$
and that the errors $\epsilon_i$ are uncorrelated (weaker than independence) and that $E(\epsilon_i) = 0$ and $Var(\epsilon_i) = \sigma^2$.
They state that they do not necessarily need to specify a probability distribution for the $Y_1, \ldots, Y_n$ (i.e. normality). Later on in the chapter, they do assume a conditional normal model, and discuss topics like the MLE of the regression estimates, etc.
Is it the case that we can get away with only the assumptions above for linear regression?

Normality
The explicit inferential conditions under which they derive MLE, and tests and confidence intervals for parameters do assume normality as you say.
Certainly if you don't have normality the ordinary least squares estimators of the coefficients are not maximum likelihood (there's a direct correspondence between the term in the exponent of the normal distribution in the usual assumptions and the least squares criterion).
It's also not hard to choose some reasonable set of other assumptions and see that the maximum likelihood estimators differ from the usual least squares estimators each time.
However, we are not forced to use maximum likelihood to estimate parameters.
Under some conditions, the least squares estimators still have many useful properties (such as being minimum variance among linear unbiased estimators).
While this is an important result we should avoid thinking of it as a panacaea; if the distribution is very far from normal, all linear unbiased estimators may be poor, so knowing you're choosing the best one may not always be very useful.
This result also relies on assumptions that are themselves often untenable. For example the constant variance assumption is in many cases unsuitable as a description of real measurements; variation related to the mean is quite common. In some kinds of data I deal with which is skewed and heteroskedastic (and has nonlinear relationships to the main predictors) I often use generalized linear models to deal with all of those issues at once. On the other hand, transformation is often adequate for the same set of issues that I tend to encounter. The main differences occur in the extreme tail of the predictions; if your main focus was on the mean you might not be too bothered by the difference.

Similarly, we may pull in the CLT plus other results to show that under certain conditions we can obtain tests and confidence intervals (but not prediction intervals in general) that should have close to the correct type I error rate and close to the correct coverage respectively, if sample sizes are 'sufficiently large' $-$ if only we knew how large would be large enough for our specific purposes.*
Again, we should take some care not to read too much into this, since significance levels and coverage are not the only things that should concern us. If the conditional distribution of the response is very non-normal, relative power may be quite poor (ARE may be arbitrarily low), and similarly with CIs the interval width may on average be very wide.

There are other paths to controlling coverage and significance level that don't rely on such arguments; for example if we know enough to choose more suitable models we can make use of those. Or we might use resampling techniques (bootstrap intervals and tests, and in some cases permutation tests). Or one may well combine both (use some more tenable assumption to obtain estimators but still use resampling to try to get better control of significance level and coverage even so). Bootstrap procedures are still 'large sample', though. In cases where they can be used exactly, permutation procedures may be quite helpful at smaller sample sizes.
In practice, it's often the case that the conditional distribution, while clearly non-normal, as long as the other population assumptions are not likely to be too far out (very nearly independent, not strongly heteroskedastic), that the properties of the usual estimators, tests and confidence intervals may be quite reasonable.

A few words on independence
Independence is trickier; the Gauss-Markov theorem relies on it for example, as do the other approaches discussed above.
Small amounts of dependence (e.g. slight serial dependence in error terms) has modest effects on things like significance levels and coverage, but substantial dependence would typically require a more suitable model to get reasonable behaviour; since dependence can take many forms, different approaches would be suitable for different problems. If there's dependence between error terms and the predictors, on the other hand, you can have bias in estimates, which can be a more serious issue.

* This relies on knowing something about the distribution (e.g. perhaps something about the size of the standardized third absolute  moment $E[|ε|^{3}]/σ^{3}$ ). If we know enough about the population to assert  such a definite fact about it, we may well know enough to choose better estimators and tests etc in the first place.
