Body: <p>I'm working in a regression setting to predict a scalar value <span class="math-container">$y$</span> from an input <span class="math-container">$\textbf{x} \in \mathbb{R}^D$</span> and <em>I'm interested in understanding whenever my model is fed with something that it is outside the (unknown) training distribution <span class="math-container">$p(\textbf{x})$</span></em>. For simplicity we can assume I'm using a simple neural network <span class="math-container">$f_\theta:\mathbb{R}^K \rightarrow\mathbb{R}$</span> to predict a single (scalar) property value, training my model with an initial dataset <span class="math-container">$\mathcal{D} = \{(\textbf{x}_i \, , y_i)\}_{i=1}^N$</span>, that is, my task is specifically about regression.</p>
  <p>What I'd be interested in achieving would be that, feeding my neural net with a new input <span class="math-container">$\tilde{\textbf{x}}$</span> I could retrieve somehow a confidence score telling me if the new input <span class="math-container">$\tilde{\textbf{x}}$</span> lies outside the spectrum of observed instances in training dataset.</p>
  <p>A way of doing that would be of course estimating the probability of training dataset <span class="math-container">$p_\theta(\textbf{x})$</span> and see if the new material <span class="math-container">$\tilde{\textbf{x}}$</span> is in a low-likelihood region of <span class="math-container">$p$</span>. People have used such approach for images (<a href="https://arxiv.org/pdf/1912.03263.pdf" rel="nofollow noreferrer">https://arxiv.org/pdf/1912.03263.pdf</a>) but generative models are hard to train.</p>
  <p>Instead, I was looking at recently proposed papers using <em>energy-scores</em> for detecting out of distribution samples (<a href="https://arxiv.org/abs/2010.03759" rel="nofollow noreferrer">paper1</a>, <a href="https://arxiv.org/pdf/2202.01197.pdf" rel="nofollow noreferrer">paper2</a>) but the examples seem to refer specifically to classification settings.</p>
  <p>As I'm not too familiar with energy-based models, is there a way such frameworks may be applied to regression settings?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-09-03T11:13:22.690Z
Comments.Text: Though I'm not familiar with energy-based methods, have you tried other "simpler" approaches? for example, assuming a mixture of Gaussians and using Mahalanobis distance as anomaly score, or trying to use kNN on the predicted values of $f_{\theta}$ to find a separating threshold based on mean distance from the neighbors?
Comments.UserDisplayName: Hadar Sharvit
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-09-03T15:16:40.813Z
Comments.Text: Many thanks for your suggestions! As I’m not so into ood I was just searching for what is out there, so I appreciate your comment a lot! Could I ask you to further elaborate on these approaches?
Comments.UserDisplayName: James Arten
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-09-03T19:03:58.923Z
Comments.Text: I would suggest going over some of the known AD surveys (here's a pretty new one here - https://arxiv.org/abs/1901.03407). If you're looking for something shorter you can check out this repo (https://github.com/Hadar933/Deep-Committee-kNN), try briefing through the introduction section.
Comments.UserDisplayName: Hadar Sharvit
Tags: <regression><multiclass-classification><energy-based-models>
Title: Out of distribution detection (OOD) in the context of regression problems
UserDisplayName: James Arten
ThreadText: # Out of distribution detection (OOD) in the context of regression problems
  
  (Asked by: username_0 on 2022-09-02)
  
  I'm working in a regression setting to predict a scalar value $y$ from an input $\textbf{x} \in \mathbb{R}^D$ and *I'm interested in understanding whenever my model is fed with something that it is outside the (unknown) training distribution $p(\textbf{x})$*. For simplicity we can assume I'm using a simple neural network $f\_\theta:\mathbb{R}^K \rightarrow\mathbb{R}$ to predict a single (scalar) property value, training my model with an initial dataset $\mathcal{D} = \{(\textbf{x}\_i \, , y\_i)\}\_{i=1}^N$, that is, my task is specifically about regression.
  
  What I'd be interested in achieving would be that, feeding my neural net with a new input $\tilde{\textbf{x}}$ I could retrieve somehow a confidence score telling me if the new input $\tilde{\textbf{x}}$ lies outside the spectrum of observed instances in training dataset.
  
  A way of doing that would be of course estimating the probability of training dataset $p\_\theta(\textbf{x})$ and see if the new material $\tilde{\textbf{x}}$ is in a low-likelihood region of $p$. People have used such approach for images (<https://arxiv.org/pdf/1912.03263.pdf>) but generative models are hard to train.
  
  Instead, I was looking at recently proposed papers using *energy-scores* for detecting out of distribution samples ([paper1](https://arxiv.org/abs/2010.03759), [paper2](https://arxiv.org/pdf/2202.01197.pdf)) but the examples seem to refer specifically to classification settings.
  
  As I'm not too familiar with energy-based models, is there a way such frameworks may be applied to regression settings?
  
  **Tags:** regression, multiclass-classification, energy-based-models
  
  > **Comments:**
  > **username_1**: Though I'm not familiar with energy-based methods, have you tried other "simpler" approaches? for example, assuming a mixture of Gaussians and using Mahalanobis distance as anomaly score, or trying to use kNN on the predicted values of $f_{\theta}$ to find a separating threshold based on mean distance from the neighbors?  
  > **username_0**: Many thanks for your suggestions! As I’m not so into ood I was just searching for what is out there, so I appreciate your comment a lot! Could I ask you to further elaborate on these approaches?  
  > **username_1**: I would suggest going over some of the known AD surveys (here's a pretty new one here - https://arxiv.org/abs/1901.03407). If you're looking for something shorter you can check out this repo (https://github.com/Hadar933/Deep-Committee-kNN), try briefing through the introduction section.  
  
  ---
  
  ## 0 answers
  
  