Bayesian Linear Regression vs Least Squares Suppose $X,Y$ are random variables and we wish to use linear regression $Y=aX+b + \epsilon$. We can determine $a,b$ by using a very straightforward least squares computation. Alternatively, we can give $a,b$ prior distributions and use Bayesian methods to find the maximum likelihoods for $a,b$ and get a linear regression that way instead. It appears that the Bayesian approach here is certainly a lot more difficult to do computationally. However, I have seen statistics books reference this method. My question is, why would someone use the Bayesian approach instead?

Dave Harris gave a detailed answer (+1), so let me just give a more general comment. You seem to worry about computational complexity, why? This should never be your biggest concerns.
In practice, for simple models such as linear regression more computationally demanding method on a modern computer would usually run few seconds, maybe minutes, slower than the less demanding one. It could make a visible difference for more complicated models where the training could take hours or days. In both cases you should however ask yourself what is most important for you for the particular problem? For example, if you have strict business requirements for the algorithm to return predictions in milliseconds, than you would worry about computational complexity and discard any demanding algorithm. On another hand, if you have a research project where in two years you need to write a report and your model would train for a week non stop, it is likely not a major problem.
Many computationally expensive algorithms have additional merits, for example, non-parametric models would not make distributional assumptions that are not appropriate for your data, obtaining uncertainty estimates as with Bayesian models, more accurate results, etc.
