Answers.Body: <p>Parametrizing something is literally what the term is saying: &quot;how do you put parameters so that you can end up having it via such parameters&quot;</p>
  <p>For example, you can parametrize the covariance with a MLP, thus the MLP &quot;spits out&quot; the covariance matrix. This way, such matrix can change for every input.</p>
  <p>However, nothing stops you from having a fixed covariance, where every entry of your matrix is a parameter (however in this way, it's fixed, and does not vary based on <span class="math-container">$x$</span>, as they do in their paper)</p>
  <p>To conclude, what they do, is to ask the neural network to output both <span class="math-container">$\mu$</span> and <span class="math-container">$\Sigma$</span> based on the current input (thus the NN parametrize the covariance)... note that later they say that they just use the diagonal</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-04-30T15:36:29.947Z
Answers.LastActivityDate: 2024-04-30T15:36:29.947Z
Answers.UserDisplayName: Alberto
Body: <p>In the paper <a href="https://arxiv.org/pdf/2112.02143" rel="nofollow noreferrer">https://arxiv.org/pdf/2112.02143</a> it is noted that there are different ways to &quot;parametrize&quot; covariance. (page 4)
  <a href="https://i.sstatic.net/pBemXqfg.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/pBemXqfg.png" alt="enter image description here" /></a></p>
  <p>What does &quot;parametrizing&quot; covariance mean exactly? In the model described in the paper, prediction uncertainty in the form of covariance is estimated using an MLP. The resulting covariance matrix is used in an uncertainty-reduction loss function to train the model. I do not really understand what different parametrizations of covariance there are. As far as I know there exists simply one covariance of model predictions.</p>
  <p>I also looked at the paper <a href="https://arxiv.org/pdf/1910.14215" rel="nofollow noreferrer">https://arxiv.org/pdf/1910.14215</a> which they reference. In it there are mentions of epistemic and aleatoric uncertainty however I do not really see how one can distinguish between the 2 when building this model, or if that is even what is meant by covariance &quot;parametrization&quot;.</p>
  
Tags: <transformer><loss><uncertainty-quantification>
Title: Understanding different methods of covariance parametrization
UserDisplayName: Homer Sanchez
ThreadText: # Understanding different methods of covariance parametrization
  
  (Asked by: username_0 on 2024-04-30)
  
  In the paper <https://arxiv.org/pdf/2112.02143> it is noted that there are different ways to "parametrize" covariance. (page 4)
  [![enter image description here](https://i.sstatic.net/pBemXqfg.png)](https://i.sstatic.net/pBemXqfg.png)
  
  What does "parametrizing" covariance mean exactly? In the model described in the paper, prediction uncertainty in the form of covariance is estimated using an MLP. The resulting covariance matrix is used in an uncertainty-reduction loss function to train the model. I do not really understand what different parametrizations of covariance there are. As far as I know there exists simply one covariance of model predictions.
  
  I also looked at the paper <https://arxiv.org/pdf/1910.14215> which they reference. In it there are mentions of epistemic and aleatoric uncertainty however I do not really see how one can distinguish between the 2 when building this model, or if that is even what is meant by covariance "parametrization".
  
  **Tags:** transformer, loss, uncertainty-quantification
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2024-04-30. Score: 0 | Accepted answer
  
  Parametrizing something is literally what the term is saying: "how do you put parameters so that you can end up having it via such parameters"
  
  For example, you can parametrize the covariance with a MLP, thus the MLP "spits out" the covariance matrix. This way, such matrix can change for every input.
  
  However, nothing stops you from having a fixed covariance, where every entry of your matrix is a parameter (however in this way, it's fixed, and does not vary based on $x$, as they do in their paper)
  
  To conclude, what they do, is to ask the neural network to output both $\mu$ and $\Sigma$ based on the current input (thus the NN parametrize the covariance)... note that later they say that they just use the diagonal
  
  
  ---
  
  