Answers.Body: <p>I think you guys are playing on semantics.</p>
  
  <p>If you consider feature extraction to be an unlearned preprocessing step to get inputs for your model, then no, word embeddings are not a feature extraction technique (examples here would be BoW counts, n-gram features, etc)</p>
  
  <p>If you consider feature extraction to be any form of conversion from text to a set of features, then yes, word embeddings should be considered a form of feature extraction, given that they've learned in the process (or stolen from another model's training). Note though if you do include this, you would probably include most pretrained models as a whole as feature extraction techniques (like BERT).</p>
  
  <p>So the whole conversation you had can go either way depending on the definitions you set.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-12-27T16:18:44.730Z
Answers.LastActivityDate: 2019-12-27T20:43:39.210Z
Answers.LastEditDate: 2019-12-27T20:43:39.210Z
Answers.UserDisplayName: mshlis
Answers.Body: <p>Though <em>word-embedding</em> is primarily a <em>language modeling tool</em>, it also acts as a feature extraction method because it helps transform raw data (characters in text documents) to a meaningful alignment of word vectors in the embedding space that the model can work with more effectively (than other traditional methods such as TF-IDF, Bag of Words, etc, on a large corpus). Word Embedding techniques help extract information from the pattern and occurence of words and goes further than other traditional token representation methods to decode/identify the meaning/context of the words, thereby providing more relevant and important features to the model to tackle the underlying problem.</p>
  
  <p>However, from another standpoint, word-embedding models were not developed aiming to solve a particular feature extraction problem, but rather, to <em>generalize</em> and <em>model</em> the language used in a corpus to gain a <em>semantic understanding</em> of the words and the <em>relationships</em> between them. Such that, all the various corpus-specific tasks can then <em>employ the same "library" of information</em> which was <em>collectively</em> &amp; <em>exhaustively</em> learnt by the embedding model. Meaning, the word embedding model learns a language model that is <em>task-agnostic</em> for all tasks on that corpus unlike feature extraction methods which are specifically task-oriented.</p>
  
  <p>Hence, the similarity is - word-embeddings can effectively aid in feature extraction; the dissimilarity is - they're not primarily meant to extract features more than they are for modeling a language which might be an "overkill" for a particular feature extraction task on a dataset.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-12-27T20:12:47.067Z
Answers.Text: One thing that **should be mentioned** is that when two semantically similar words are mapped into the *embedding space*, the defined metric within the space (for word2vec it is cosine distance) is proportional to the word similarity.  For example 'hog' and 'pig' will have a cosine distance(distance of 1 means same) closer to 1 than would 'hog' and 'cow'.  So neighbours in the embedding space are similar in the language. I consider this property to be the most important aspect of embedding.
Answers.UserDisplayName: user1269942
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-12-27T17:14:12.260Z
Answers.LastActivityDate: 2019-12-27T17:19:16.203Z
Answers.LastEditDate: 2019-12-27T17:19:16.203Z
Answers.UserDisplayName: Balraj Ashwath
Body: <p>Feature extraction is a concept concerning the translation of raw data into the inputs that a particular machine learning algorithm requires. These derived features from the raw data that are actually relevant to tackle the underlying problem. On the other hand, word embeddings are basically distributed representations of text in an n-dimensional space.</p>
  
  <p>As far as I understand, word embedding is a somehow feature extraction technique. Am I wrong ? I had an argument with a friend who believes the two topics are totally separate. Is he right? What are the similarities and dissimilarities between word embedding and feature extraction?</p>
  
Tags: <comparison><word-embedding><feature-extraction>
Title: Is word embedding a form of feature extraction?
UserDisplayName: HiDDeN
ThreadText: # Is word embedding a form of feature extraction?
  
  (Asked by: username_0 on 2019-12-27)
  
  Feature extraction is a concept concerning the translation of raw data into the inputs that a particular machine learning algorithm requires. These derived features from the raw data that are actually relevant to tackle the underlying problem. On the other hand, word embeddings are basically distributed representations of text in an n-dimensional space.
  
  As far as I understand, word embedding is a somehow feature extraction technique. Am I wrong ? I had an argument with a friend who believes the two topics are totally separate. Is he right? What are the similarities and dissimilarities between word embedding and feature extraction?
  
  **Tags:** comparison, word-embedding, feature-extraction
  
  ---
  
  ## 2 answers
  
  ### Answer by username_2 on 2019-12-27. Score: 5 | Accepted answer
  
  Though *word-embedding* is primarily a *language modeling tool*, it also acts as a feature extraction method because it helps transform raw data (characters in text documents) to a meaningful alignment of word vectors in the embedding space that the model can work with more effectively (than other traditional methods such as TF-IDF, Bag of Words, etc, on a large corpus). Word Embedding techniques help extract information from the pattern and occurence of words and goes further than other traditional token representation methods to decode/identify the meaning/context of the words, thereby providing more relevant and important features to the model to tackle the underlying problem.
  
  However, from another standpoint, word-embedding models were not developed aiming to solve a particular feature extraction problem, but rather, to *generalize* and *model* the language used in a corpus to gain a *semantic understanding* of the words and the *relationships* between them. Such that, all the various corpus-specific tasks can then *employ the same "library" of information* which was *collectively* & *exhaustively* learnt by the embedding model. Meaning, the word embedding model learns a language model that is *task-agnostic* for all tasks on that corpus unlike feature extraction methods which are specifically task-oriented.
  
  Hence, the similarity is - word-embeddings can effectively aid in feature extraction; the dissimilarity is - they're not primarily meant to extract features more than they are for modeling a language which might be an "overkill" for a particular feature extraction task on a dataset.
  
  > **Comments:**
  > **username_3**: One thing that **should be mentioned** is that when two semantically similar words are mapped into the *embedding space*, the defined metric within the space (for word2vec it is cosine distance) is proportional to the word similarity.  For example 'hog' and 'pig' will have a cosine distance(distance of 1 means same) closer to 1 than would 'hog' and 'cow'.  So neighbours in the embedding space are similar in the language. I consider this property to be the most important aspect of embedding.  
  
  ---
  
  ### Answer by username_1 on 2019-12-27. Score: 2
  
  I think you guys are playing on semantics.
  
  If you consider feature extraction to be an unlearned preprocessing step to get inputs for your model, then no, word embeddings are not a feature extraction technique (examples here would be BoW counts, n-gram features, etc)
  
  If you consider feature extraction to be any form of conversion from text to a set of features, then yes, word embeddings should be considered a form of feature extraction, given that they've learned in the process (or stolen from another model's training). Note though if you do include this, you would probably include most pretrained models as a whole as feature extraction techniques (like BERT).
  
  So the whole conversation you had can go either way depending on the definitions you set.
  
  
  ---
  
  