Inference: How is the Laplace approximation actually useful to us compared with MLE and MAP? I was reading a few different sources (including the "Machine Learning and Pattern Recognition" book by Bishop) about the Laplace integral approximation method for inference. However, I am still confused about the use of this method within inference.
Question: In what sense is the Laplace approximation better than MAP and MLE for inference calculations? Why would the enabling of the integration allow for better results?
Attempt: My understanding is as follows based on the reading:
In working probabilistically with parameters, we start by writing the probability of everything, also called the generative model. Then we get:
$$ p(f_{\star} | \mathcal{D}) = \frac{p(f_{\star}, \mathcal{D})}{p(\mathcal{D})} = \frac{\int p(f_{\star}, \mathcal{D}, \theta) d\theta}{p(\mathcal{D})} = \frac{\int p(f_{\star}| \mathcal{D}, \theta) p(\mathcal{D} | \theta) p(\theta) d\theta}{p(\mathcal{D})} $$
(However, I am not quite sure what $ f_{\star} $ is).
Then, the literature goes on to say that the core challenge of probabilistic inference is integration, marginalizing over the unknown, e.g
$$ p(f_{\star} | \mathcal{D}) = \int p(f_{\star} | \mathcal{D}, \theta) p(\theta | \mathcal{D}) d\theta $$
and
$$ p(\mathcal{D}) = \int p(\mathcal{D} | \theta) p(\theta) d \theta $$
Marginalizing can be interpreted as averaging over possible $\theta$ values, weighted by posterior probabilities (is this referring to the first or second integral?). Remember that maximum likelihood aimed to maximize $p(\mathcal{D} | \theta)$ and MAP aimed to maximize $p(\theta | \mathcal{D})$. These approximated the aforementioned expressions by delta functions to resolve tractable integrals. Then the reading says: "Better alternatives come up with more accurate models of those pdfs that nonetheless allow integration to be performed."
[EDIT #1]: I understand the mathematics of the method (i.e. using Taylor approximation to find mode of distribution we want to estimate; fitting a gaussian to approximate the function which then allows us to use standard results to compute the integral), but I just don't understand what the point of that is (basically a 'so what?' question)

You are comparing apples and bicycles ... mle and map are estimation methods/principles, Laplace approximation is a particular mathematical approximation used when the likelihood function is difficult to deal with directly. Perusing some of the answers tagged with laplace-approximation should convince you of its usefulness!
One textbook with a good treatment of the use of Laplace Approximation is Statistical Models by  A. C. Davison. Have a look.  But the main point is to ask yourself: What to do when you have an intractable likelihood function?
