Making sure that the design matrix is positive (semi-) definite In Bayesian linear regression, how do I make sure that the design matrix produced by a neural network $ \Phi$ is positive definite?
Computing the covariance matrix on the weight requires inverting --- i.e., $(\Phi^\text{T}\Phi)^{-1}$.  For reference, the posterior on the weights follows: $\theta \sim \mathcal{N}(m_N,S_N)$ and prior  $\theta \sim \mathcal{N}(0,\alpha^{-1}I)$
$S_N^{-1} = \beta \cdot (\Phi^T\Phi)^{-1} + \alpha \cdot I$ where $\beta$ is the noise precision on output.
$m_N = \beta \cdot S_N \cdot \Phi^\text{T} \cdot t$ where $t$ is the target on seen data.

If you want $(\Phi^T\Phi)^{-1}$ to be invertible (where $\Phi \in R^{N \times K})$ you need to ensure $rk(\Phi) = K$ (i.e. full rank). As mentioned in the comments, if $\Phi$ is rank deficient then $\Phi^T\Phi$ is positive semidefinite and thus not always invertible.
