Answers.Body: <p>What I did when I encountered sparse inputs was to preprocess the data to only include vectors with more than n (threshold) nonzero values.</p>
  <p>Not really an elegant solution (since you give up a huge part of your data), but you can try tweaking the threshold for best results.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-04-06T02:31:38.953Z
Answers.LastActivityDate: 2022-04-06T02:31:38.953Z
Answers.UserDisplayName: DKDK
Answers.Body: <p>The default solution to handle sparse or one-hot encoded input data is to use embeddings(<a href="https://pytorch.org/docs/stable/generated/torch.nn.Embedding.html" rel="nofollow noreferrer">PyTorch Embeddings</a>).
  The binary, n-dimensional input data is then transformed to a small dimensional(e.g., 4-128 dim) vector containing real values.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-08-25T12:41:54.510Z
Answers.LastActivityDate: 2022-08-25T12:41:54.510Z
Answers.UserDisplayName: a1o
Answers.Body: <p>I'm not allowed to leave comments so I will put it here. I would use some sort of dimensionality reduction algo, PCA or something more sophisticated if need be. This way you can reduce your input to a smaller number of dimensions, which are more easy to handle</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-01-29T14:00:10.863Z
Answers.LastActivityDate: 2024-01-29T14:01:03.720Z
Answers.LastEditDate: 2024-01-29T14:01:03.720Z
Answers.UserDisplayName: ArtemJDS
Body: <p>I have a task I want to solve with neural networks. The task is predicting a certain vector of dimension K. The problem is that the inputs to the networks are sparse.</p>
  <p>The input is a vector of size N, where N is huge (&gt; 1M) and for most cases, the vast majority of the entries (&gt; 99%) in the input vector are 0. Very rare examples however do have almost full inputs. It's clear that the model is hard to train, as there might be huge weight imbalance within examples, while the target vectors need not have such an imbalance.</p>
  <p>I do have a working model, but could you point me to some papers / relevant literature about training a network whose inputs are so sparse? Perhaps there are some techniques that could be useful (maybe some preprocessing steps on the inputs, or something along those lines).</p>
  <p>Any hint is appreciated!</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-04-06T10:37:22.347Z
Comments.Text: Can you please put your **specific** question in the title? "Neural networks with sparse inputs" is not a question and it's also not specific enough.
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-04-22T07:12:16.107Z
Comments.Text: What is the percentage of fully populated inputs? Can they be thrown away? Did you try usual dimensionallity reduction techniques? Did you try embeddings like in language processing? What else did you try? What kind of network is your working model?
Comments.UserDisplayName: Vladislav Gladkikh
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-01-25T16:40:56.893Z
Comments.Text: I am not an expert in sparse computation, but [this Pytorch tutorial](https://pytorch.org/docs/stable/sparse.html) might be a great start.
Comments.UserDisplayName: Minh-Long Luu
Tags: <neural-networks><input-layer>
Title: Neural networks with sparse inputs
UserDisplayName: Ant
ThreadText: # Neural networks with sparse inputs
  
  (Asked by: username_0 on 2022-04-04)
  
  I have a task I want to solve with neural networks. The task is predicting a certain vector of dimension K. The problem is that the inputs to the networks are sparse.
  
  The input is a vector of size N, where N is huge (> 1M) and for most cases, the vast majority of the entries (> 99%) in the input vector are 0. Very rare examples however do have almost full inputs. It's clear that the model is hard to train, as there might be huge weight imbalance within examples, while the target vectors need not have such an imbalance.
  
  I do have a working model, but could you point me to some papers / relevant literature about training a network whose inputs are so sparse? Perhaps there are some techniques that could be useful (maybe some preprocessing steps on the inputs, or something along those lines).
  
  Any hint is appreciated!
  
  **Tags:** neural-networks, input-layer
  > **Comments:**
  > **username_1**: Can you please put your **specific** question in the title? "Neural networks with sparse inputs" is not a question and it's also not specific enough.  
  > **username_2**: What is the percentage of fully populated inputs? Can they be thrown away? Did you try usual dimensionallity reduction techniques? Did you try embeddings like in language processing? What else did you try? What kind of network is your working model?  
  > **username_3**: I am not an expert in sparse computation, but [this Pytorch tutorial](https://pytorch.org/docs/stable/sparse.html) might be a great start.  
  
  ---
  
  ## 3 answers
  
  ### Answer by username_4 on 2022-04-06. Score: 1
  
  What I did when I encountered sparse inputs was to preprocess the data to only include vectors with more than n (threshold) nonzero values.
  
  Not really an elegant solution (since you give up a huge part of your data), but you can try tweaking the threshold for best results.
  
  
  ---
  
  ### Answer by username_5 on 2022-08-25. Score: 0
  
  The default solution to handle sparse or one-hot encoded input data is to use embeddings([PyTorch Embeddings](https://pytorch.org/docs/stable/generated/torch.nn.Embedding.html)).
  The binary, n-dimensional input data is then transformed to a small dimensional(e.g., 4-128 dim) vector containing real values.
  
  
  ---
  
  ### Answer by username_6 on 2024-01-29. Score: 0
  
  I'm not allowed to leave comments so I will put it here. I would use some sort of dimensionality reduction algo, PCA or something more sophisticated if need be. This way you can reduce your input to a smaller number of dimensions, which are more easy to handle
  
  
  ---
  
  