How does maximizing ELBO in Bayesian neural networks give us the correct posterior predictive distribution? In Bayesian/variational neural networks one often uses the Evidence Lower BOund (ELBO) as the objective function to optimize with respect to the model parameters. That is if $D=\{y_i,x_i\}_{1\dots n}$ is the set of our training data, $\{y,x\}$ is a new data point, and $w$ represents our network weights/parameters, then the posterior predictive distribution is:
\begin{array}{rl}
 p(y|x,D) =& \int p(y|x,w,D)p(w|x,D)dw \\
=& \int p(y|x,w)p(w|D)dw \\
\end{array}
We don't know $p(w|D)$ so we instead approximate it with some known parameteric function (always seemingly chosen to be a Gaussian density) which we call $q_\phi(w)$. We want $q_\phi(w)$ to be close to $p(w|D)$ so we minimize the KL divergence between them as part of our optimization procedure. That is we end up with $d_{KL}(q_\phi(w)||p(w|D))$ in our objective function, essentially, in addition to the usual likelihood component. Thus the ELBO is:
$$E_{q_\phi(w)}[log(p(D|w))] - d_{KL}(q_\phi(w)||p(w|D))$$
Here is my question: how does maximizing ELBO lead to a good/correct posterior predictive distribution $p(y|x,D)$? The first term $$E_{q_\phi(w)}[log(p(D|w))]$$ ensures that our model accurately models the mean of the posterior predictive distribution, while the second term ensures that the approximate posterior of the weight (note this is not the posterior of $y$!) distribution is close to its true distribution. Why do we care that the approximate posterior  of the weight distribution is good? I don't see how using our objective function to approximate $p(w|D)$ well translates to $p(y|x,D)$ also being well approximated.

With Bayesian methods, in general, you extract information from your data $D$ which, together with some prior belief, leads to a posterior for your parameters $w$. That means this posterior $p(w|D)$ is your best guess for a distribution over $w$.
And if you would now generate new data $(x, y)$ by sampling a $w$ from $q_\phi(w)$, and then sampling for a given $x$ a $y$ from $p(y|x, w)$ you would get pairs $(x, y)$ that, if all your prior belief is right, would belong to the same population as your previous data $D$.
