Body: <p>I've lot of training data points (i.e in millions) and I've around few features but the issue with that is all the features are categorical data with 1 million+ categories in each.</p>
  
  <p>So, I couldn't use one hot encoding because it's not efficient so I went with the other option which is embedding of fixed length. I've just used neural nets to compute embedding.</p>
  
  <p>My question is can we use advanced NLP models like bert to extract embeddings for categorical data from my corpus? Is it possible? I've only asked it because I've only heard that bert is good for sentence embeddings.</p>
  
  <p>Thank you.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-01-20T06:20:17.453Z
Comments.Text: Are your categorical features words/sentences? Why do you want to embed categorical features?
Comments.UserDisplayName: Sharan
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-01-20T19:06:09.643Z
Comments.Text: @Sharan It's just words not sentences. There are lot of categories and if I choose to use one hot encoding it would result in a extremely sparse data-set. So only other option is to use embeddings.
Comments.UserDisplayName: user_12
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-01-21T13:05:21.777Z
Comments.Text: Yes I am not suggesting one hot. I just wanted to know how you arrived at embeddings. BERT is very computationally expensive and it would be inefficient to use it for "1 million+" categories. I d suggest you to start off with something simple like Word2Vec. Also simple mathematical features might help in some case - https://www.analyticsvidhya.com/blog/2018/02/the-different-methods-deal-text-data-predictive-python/
Comments.UserDisplayName: Sharan
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-01-21T13:48:50.293Z
Comments.Text: @Sharan I understand but I figured bert can understand the context better than any simple methods like word2vec etc.. So was just wondering if it could work on categories?
Comments.UserDisplayName: user_12
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-01-28T19:36:19.330Z
Comments.Text: This question needs more context. What exactly is the problem you are trying to solve? What are training data, what are features, what are the categories? Please clarify.
Comments.UserDisplayName: Mathias Müller
Tags: <neural-networks><machine-learning><deep-learning><natural-language-processing><bert>
Title: Can Bert be used to extract embedding for large categorical features?
UserDisplayName: user_12
ThreadText: # Can Bert be used to extract embedding for large categorical features?
  
  (Asked by: username_0 on 2020-01-19)
  
  I've lot of training data points (i.e in millions) and I've around few features but the issue with that is all the features are categorical data with 1 million+ categories in each.
  
  So, I couldn't use one hot encoding because it's not efficient so I went with the other option which is embedding of fixed length. I've just used neural nets to compute embedding.
  
  My question is can we use advanced NLP models like bert to extract embeddings for categorical data from my corpus? Is it possible? I've only asked it because I've only heard that bert is good for sentence embeddings.
  
  Thank you.
  
  **Tags:** neural-networks, machine-learning, deep-learning, natural-language-processing, bert
  
  > **Comments:**
  > **username_1**: Are your categorical features words/sentences? Why do you want to embed categorical features?  
  > **username_0**: @username_1 It's just words not sentences. There are lot of categories and if I choose to use one hot encoding it would result in a extremely sparse data-set. So only other option is to use embeddings.  
  > **username_1**: Yes I am not suggesting one hot. I just wanted to know how you arrived at embeddings. BERT is very computationally expensive and it would be inefficient to use it for "1 million+" categories. I d suggest you to start off with something simple like Word2Vec. Also simple mathematical features might help in some case - https://www.analyticsvidhya.com/blog/2018/02/the-different-methods-deal-text-data-predictive-python/  
  > **username_0**: @username_1 I understand but I figured bert can understand the context better than any simple methods like word2vec etc.. So was just wondering if it could work on categories?  
  > **username_2**: This question needs more context. What exactly is the problem you are trying to solve? What are training data, what are features, what are the categories? Please clarify.  
  
  ---
  
  ## 0 answers
  
  