Body: <p><a href="https://i.sstatic.net/NWgsJ.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/NWgsJ.png" alt="picture" /></a>
  The picture shows the model I want to implement. As you can see, the input will be a single sentence (e.g. I want to massacre those ugly people!). Then, the teacher model will output weights(which will be 0~1 float values) to more important words to classify (e.g. The model will give more weights to words like massacre and ugly for classifying the sentence to be hate-speech). Then, the student model takes those weights from the teacher and the same sentence, and the weights will be a hint for student to know which words it should focus and classifies the sentence.</p>
  <p>What I want to know is that how can I make student model to focus several words than others? Just multiplying weights to the word embeddings naively doesn't seem a good solution I think. Please give me some insight. Thank you.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-12-23T10:52:35.720Z
Comments.Text: it seems to me like the first model would just learn the equivalent of an attention mask over a sentence. So my question is, why training two models instead of a single one with attention layers? It sound like you're making your life harder this way.
Comments.UserDisplayName: Edoardo Guerriero
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-12-23T10:57:56.857Z
Comments.Text: @Edoardo Guerriero Label of first model will be the LIME values which are precomputed so that teacher model can mimic LIME values of one sentence. Would it be more helpful if I just use attention layer in the student model instead of using such LIME value?
Comments.UserDisplayName: Minimum
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-12-23T12:18:25.920Z
Comments.Text: if the LIME you're referring to is the algorithm to infer feature importance from an already trained classifier than yes, I would skip that part completely and train a model which include attention layers in its architecture from scratch. First reason: if you have access to the LIME weights of an already trained model then you already have a trained model, I don't see why you should spend time training a model that emulates another one, especially considering that you can't get better performances due to error propagation.
Comments.UserDisplayName: Edoardo Guerriero
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-12-23T12:19:37.813Z
Comments.Text: Second reason: LIME is an interpretation algorithm, using it to generate labels is a huge conceptual stretch. But maybe I misunderstood your intention, I think you should at the very least add the LIME part to the question to make it more clear.
Comments.UserDisplayName: Edoardo Guerriero
Tags: <natural-language-processing>
Title: How can I put more weight specific words when doing text classification?
UserDisplayName: Minimum
ThreadText: # How can I put more weight specific words when doing text classification?
  
  (Asked by: username_0 on 2022-12-23)
  
  [![picture](https://i.sstatic.net/NWgsJ.png)](https://i.sstatic.net/NWgsJ.png)
  The picture shows the model I want to implement. As you can see, the input will be a single sentence (e.g. I want to massacre those ugly people!). Then, the teacher model will output weights(which will be 0~1 float values) to more important words to classify (e.g. The model will give more weights to words like massacre and ugly for classifying the sentence to be hate-speech). Then, the student model takes those weights from the teacher and the same sentence, and the weights will be a hint for student to know which words it should focus and classifies the sentence.
  
  What I want to know is that how can I make student model to focus several words than others? Just multiplying weights to the word embeddings naively doesn't seem a good solution I think. Please give me some insight. Thank you.
  
  **Tags:** natural-language-processing
  
  > **Comments:**
  > **username_1**: it seems to me like the first model would just learn the equivalent of an attention mask over a sentence. So my question is, why training two models instead of a single one with attention layers? It sound like you're making your life harder this way.  
  > **username_0**: @username_1 Label of first model will be the LIME values which are precomputed so that teacher model can mimic LIME values of one sentence. Would it be more helpful if I just use attention layer in the student model instead of using such LIME value?  
  > **username_1**: if the LIME you're referring to is the algorithm to infer feature importance from an already trained classifier than yes, I would skip that part completely and train a model which include attention layers in its architecture from scratch. First reason: if you have access to the LIME weights of an already trained model then you already have a trained model, I don't see why you should spend time training a model that emulates another one, especially considering that you can't get better performances due to error propagation.  
  > **username_1**: Second reason: LIME is an interpretation algorithm, using it to generate labels is a huge conceptual stretch. But maybe I misunderstood your intention, I think you should at the very least add the LIME part to the question to make it more clear.  
  
  ---
  
  ## 0 answers
  
  