Answers.Body: <p>You may want to take a look at <a href="https://towardsdatascience.com/identifying-the-right-meaning-of-the-words-using-bert-817eef2ac1f0" rel="nofollow noreferrer">this article</a>, but I'll summarize. You can use BERT (or some other tool) to make embeddings of every word in every sentence. Then for each word, make a contextualized embedding vector using the rest of the sentence. <a href="https://github.com/negedng/bert-embedding/tree/master/bert_embedding" rel="nofollow noreferrer">bert-embedding</a> does all of this itself. Then keep the embedding vector for the important words.</p>
  <p>For each important word, you would then have two pieces of information: the embedding vector and the correct label (which could easily be made into an integer from <span class="math-container">$0$</span> to <span class="math-container">$4$</span>). Depending on the size of the embedding vectors, you could use PCA to reduce the size, although this may not be needed. Using this data, you can then train a neural network or use a k-nearest neighbors classifier.</p>
  <p>There is more information in the article, which I suggest that you read. They do a better job explaining than me and also have some actual code you may want to look at.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-08-09T02:01:47.573Z
Answers.LastActivityDate: 2020-08-09T02:11:21.803Z
Answers.LastEditDate: 2020-08-09T02:11:21.803Z
Answers.UserDisplayName: Varun Vejalla
Body: <p>I am somewhat a novice at the topic of Neural Netoworks and PyTorch.</p>
  <p>I am trying to create a model that takes a word (that I have modified very slightly) and a 'window' of context around it and predicts one of 5 tags (the tags relate to what sort of action I should perform on that word to get its correct form).</p>
  <p>For example, here's what I would call a window of size 7 and it's tag (what it means isn't too important, it's just the 'target'):</p>
  <pre><code>        Sentence                    Label
  here is a sentence for my network     N
  </code></pre>
  <p><code>sentence</code> is the word that I want the network to predict the label for, but the 3 words on either side provide contextual meaning. My problem is, how would I get a network to know I want it to predict for that central word but not outright ignore the others? I am familiar with more normal NLP tasks such as NMT and character level classification.</p>
  <p>I have already gotten my dataset 'padded' out so they're all of equal size.</p>
  <p>Any help is appreciated</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-08-08T23:51:46.200Z
Comments.Text: I don't know how to do this specifically with pytorch, but for a recent project, we used BERT to make embeddings for the sentences, and then used a k-neighbors classifier to classify the words.
Comments.UserDisplayName: Varun Vejalla
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-08-09T01:06:19.027Z
Comments.Text: @VarunVejalla So will that basically focus on my 'centre' word while taking the others into account? Thanks for the help also!
Comments.UserDisplayName: Justin Cunningham
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-08-09T02:02:19.403Z
Comments.Text: You're welcome! That's pretty much right. I explained more in my answer.
Comments.UserDisplayName: Varun Vejalla
Tags: <neural-networks><natural-language-processing><python><pytorch>
Title: Get Neural Network to predict a tag/class on a certain word using the surrounding words as context [PyTorch]?
UserDisplayName: Justin Cunningham
ThreadText: # Get Neural Network to predict a tag/class on a certain word using the surrounding words as context [PyTorch]?
  
  (Asked by: username_0 on 2020-08-08)
  
  I am somewhat a novice at the topic of Neural Netoworks and PyTorch.
  
  I am trying to create a model that takes a word (that I have modified very slightly) and a 'window' of context around it and predicts one of 5 tags (the tags relate to what sort of action I should perform on that word to get its correct form).
  
  For example, here's what I would call a window of size 7 and it's tag (what it means isn't too important, it's just the 'target'):
  
  ```
          Sentence                    Label
  here is a sentence for my network     N
  
  ```
  
  `sentence` is the word that I want the network to predict the label for, but the 3 words on either side provide contextual meaning. My problem is, how would I get a network to know I want it to predict for that central word but not outright ignore the others? I am familiar with more normal NLP tasks such as NMT and character level classification.
  
  I have already gotten my dataset 'padded' out so they're all of equal size.
  
  Any help is appreciated
  
  **Tags:** neural-networks, natural-language-processing, python, pytorch
  
  > **Comments:**
  > **username_1**: I don't know how to do this specifically with pytorch, but for a recent project, we used BERT to make embeddings for the sentences, and then used a k-neighbors classifier to classify the words.  
  > **username_0**: @username_1 So will that basically focus on my 'centre' word while taking the others into account? Thanks for the help also!  
  > **username_1**: You're welcome! That's pretty much right. I explained more in my answer.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2020-08-09. Score: 1 | Accepted answer
  
  You may want to take a look at [this article](https://towardsdatascience.com/identifying-the-right-meaning-of-the-words-using-bert-817eef2ac1f0), but I'll summarize. You can use BERT (or some other tool) to make embeddings of every word in every sentence. Then for each word, make a contextualized embedding vector using the rest of the sentence. [bert-embedding](https://github.com/negedng/bert-embedding/tree/master/bert_embedding) does all of this itself. Then keep the embedding vector for the important words.
  
  For each important word, you would then have two pieces of information: the embedding vector and the correct label (which could easily be made into an integer from $0$ to $4$). Depending on the size of the embedding vectors, you could use PCA to reduce the size, although this may not be needed. Using this data, you can then train a neural network or use a k-nearest neighbors classifier.
  
  There is more information in the article, which I suggest that you read. They do a better job explaining than me and also have some actual code you may want to look at.
  
  
  ---
  
  