Answers.Body: <p>I don't fully agree with the other answer, mainly on point 1.</p>
  <p>First thing first, adding a &quot;nothing&quot; label, means that you also have to gather new data to train that &quot;nothing&quot; neuron.</p>
  <p>Instead, in my opinion, you want a density estimation filter at the beginning of your process, which would mean to have a 2 stage pipeline:</p>
  <ol>
  <li>check if the incoming data comes from your training dataset</li>
  <li>if so, classify it</li>
  </ol>
  <p>The nice part of the density estimation, is that you don't need any additional data to train it, and the first model that comes in my mind for this, is VAE.</p>
  <p>What you want to do with VAE is this:</p>
  <ol>
  <li>train the VAE</li>
  <li>do a grid search on which KL divergence it's better to use as a threshold</li>
  <li>deploy VAE and classifier, and use the VAE as a first stage filtering</li>
  </ol>
  <p>In addition to this, the <em>You can try to do preprocessing</em> i don't really see how it could help.</p>
  <p>About the <em>You may also want to estimate your model's uncertainty</em>... if you use the entropy as he is explaining on his blogpost, you are relying on the assumption that the model is well calibrated, which means that you should have a model that predicts a uniform distribution if confused... however, even though this is theoretically meaningful, a neural network might or might not do that.</p>
  <p>For example, think about if you only have 2 samples, 1 per class, binary classification. Any boundary boundary between the two can have a 0 loss, however what you want, is to have something like a SVM boundary, so that the space is evenly split between the two points (which is not in any way guaranteed or even encouraged to be done by a NN)<br />
  Alternatively, you can use Bayesian NN, but they also have some practical biases when implemented, so they don't work that well...</p>
  <p>You want probably the simple/most performing model uncertainty technique? Use ensembles</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-08-16T21:47:21.700Z
Answers.Text: I will give a shot to VAE now and let you know how it goes. I did try adding a noise dataset but the results were so-so.
Answers.UserDisplayName: Sterling Duchess
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-08-16T23:00:35.980Z
Answers.Text: If I understand you correctly you are suggesting a two layered model approach. To train both a VAE and my Conv1D classification model. Then first pass the data to VAE to remove remove the noise or to try to construct one of the 12 unique gestures than pass the output of VAE to classification model for prediction ?
Answers.UserDisplayName: Sterling Duchess
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-08-16T23:06:54.447Z
Answers.Text: @SterlingDuchess idk what you mean with "2 layer", usually it's called "2 stage approach"... the VAE is used to assess if it's an outlier (one of those random events that you don't really care) and then if it's not, to use a classifier to classify it
Answers.UserDisplayName: Alberto
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-08-17T08:30:27.070Z
Answers.Text: I mean basically have two model definitions one VAE and my softmax model and first pass the input to VAE then to Softmax or ? Do you maybe have an example I never worked with autoencoders before.
Answers.UserDisplayName: Sterling Duchess
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-08-17T23:06:34.240Z
Answers.Text: @SterlingDuchess look up "anomaly detection with VAE", and then the code is like `if vae_encoder.predict(data).norm() > threshold: return classifier(data) else: None`
Answers.UserDisplayName: Alberto
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-08-22T15:00:14.800Z
Answers.Text: Alberto I just wanted to thank you for the suggestion it really worked very well. However I went with a regular AE but I will try a VAE model later on. Its very very fast and it also resolves another unrelated issue for me. I added a AE, I run a encode/decode then compare the cosine+temporal distance between given sample and reconstruction and if its above a threshold I classify it.
Answers.UserDisplayName: Sterling Duchess
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-08-14T22:46:37.230Z
Answers.LastActivityDate: 2023-08-14T22:46:37.230Z
Answers.UserDisplayName: Alberto
Body: <p>I have a device with an accelerometer and gyroscope (6-axis). The device sends live raw telemetry data to the model 40 samples for each input, 6 values per sample (accelerometer xyz, gyroscope xyz). The model predicts between 12 different labels of unique motions, the dataset has a size of 120k x 40 x 6, ~10k/label.</p>
  <p>It's not a small dataset but at 10k/label it's also not too big. I have a Conv1D model that ends with a Softmax layer and it works extremely well. I make sure to shuffle my training/test dataset for more variance before splitting it 80:20.</p>
  <p>I get 0.998 validation accuracy and 0.031 validation loss. And in live application it performs well.</p>
  <p><strong>However,</strong> at times (quite often) motions that are just noise for instance person walking around or just waving the device around is predicted as one of these 12 unique labels with high probability due to Softmax since that one happens to be the closest to the input data but is in fact very different.</p>
  <p><strong>How,</strong> do you deal with this ? One option I see it to just train the model with at 13th label being basically NOTHING and recording noise. But there has to be a better solution.</p>
  <p>I have also tried sigmoid and binary crossentropy but the dataset is small for this kind of application and I only get about ~ 0.5/0.84 on validation loss/accuracy which is not really usable.</p>
  <p>In this case I would have to drastically increase the dataset.</p>
  
Tags: <convolutional-neural-networks><tensorflow><softmax>
Title: Dealing with noise in models with softmax output
UserDisplayName: Sterling Duchess
ThreadText: # Dealing with noise in models with softmax output
  
  (Asked by: username_0 on 2023-08-13)
  
  I have a device with an accelerometer and gyroscope (6-axis). The device sends live raw telemetry data to the model 40 samples for each input, 6 values per sample (accelerometer xyz, gyroscope xyz). The model predicts between 12 different labels of unique motions, the dataset has a size of 120k x 40 x 6, ~10k/label.
  
  It's not a small dataset but at 10k/label it's also not too big. I have a Conv1D model that ends with a Softmax layer and it works extremely well. I make sure to shuffle my training/test dataset for more variance before splitting it 80:20.
  
  I get 0.998 validation accuracy and 0.031 validation loss. And in live application it performs well.
  
  **However,** at times (quite often) motions that are just noise for instance person walking around or just waving the device around is predicted as one of these 12 unique labels with high probability due to Softmax since that one happens to be the closest to the input data but is in fact very different.
  
  **How,** do you deal with this ? One option I see it to just train the model with at 13th label being basically NOTHING and recording noise. But there has to be a better solution.
  
  I have also tried sigmoid and binary crossentropy but the dataset is small for this kind of application and I only get about ~ 0.5/0.84 on validation loss/accuracy which is not really usable.
  
  In this case I would have to drastically increase the dataset.
  
  **Tags:** convolutional-neural-networks, tensorflow, softmax
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2023-08-14. Score: 2 | Accepted answer
  
  I don't fully agree with the other answer, mainly on point 1.
  
  First thing first, adding a "nothing" label, means that you also have to gather new data to train that "nothing" neuron.
  
  Instead, in my opinion, you want a density estimation filter at the beginning of your process, which would mean to have a 2 stage pipeline:
  
  1. check if the incoming data comes from your training dataset
  2. if so, classify it
  
  The nice part of the density estimation, is that you don't need any additional data to train it, and the first model that comes in my mind for this, is VAE.
  
  What you want to do with VAE is this:
  
  1. train the VAE
  2. do a grid search on which KL divergence it's better to use as a threshold
  3. deploy VAE and classifier, and use the VAE as a first stage filtering
  
  In addition to this, the *You can try to do preprocessing* i don't really see how it could help.
  
  About the *You may also want to estimate your model's uncertainty*... if you use the entropy as he is explaining on his blogpost, you are relying on the assumption that the model is well calibrated, which means that you should have a model that predicts a uniform distribution if confused... however, even though this is theoretically meaningful, a neural network might or might not do that.
  
  For example, think about if you only have 2 samples, 1 per class, binary classification. Any boundary boundary between the two can have a 0 loss, however what you want, is to have something like a SVM boundary, so that the space is evenly split between the two points (which is not in any way guaranteed or even encouraged to be done by a NN)  
  Alternatively, you can use Bayesian NN, but they also have some practical biases when implemented, so they don't work that well...
  
  You want probably the simple/most performing model uncertainty technique? Use ensembles
  
  > **Comments:**
  > **username_0**: I will give a shot to VAE now and let you know how it goes. I did try adding a noise dataset but the results were so-so.  
  > **username_0**: If I understand you correctly you are suggesting a two layered model approach. To train both a VAE and my Conv1D classification model. Then first pass the data to VAE to remove remove the noise or to try to construct one of the 12 unique gestures than pass the output of VAE to classification model for prediction ?  
  > **username_1**: @SterlingDuchess idk what you mean with "2 layer", usually it's called "2 stage approach"... the VAE is used to assess if it's an outlier (one of those random events that you don't really care) and then if it's not, to use a classifier to classify it  
  > **username_0**: I mean basically have two model definitions one VAE and my softmax model and first pass the input to VAE then to Softmax or ? Do you maybe have an example I never worked with autoencoders before.  
  > **username_1**: @SterlingDuchess look up "anomaly detection with VAE", and then the code is like `if vae_encoder.predict(data).norm() > threshold: return classifier(data) else: None`  
  > **username_0**: Alberto I just wanted to thank you for the suggestion it really worked very well. However I went with a regular AE but I will try a VAE model later on. Its very very fast and it also resolves another unrelated issue for me. I added a AE, I run a encode/decode then compare the cosine+temporal distance between given sample and reconstruction and if its above a threshold I classify it.  
  
  ---
  
  