Answers.Body: <p>Yes, you are doing it right.</p>
  <p>There is an implementation of MNLP here: <a href="https://github.com/VectorInstitute/DANER/blob/2b5feadda6c1d35ab7be09277339c490396bc49e/backend/models/al_model.py#L193" rel="nofollow noreferrer">https://github.com/VectorInstitute/DANER/blob/2b5feadda6c1d35ab7be09277339c490396bc49e/backend/models/al_model.py#L193</a></p>
  <p>It looks like it is the greedy approximation: <span class="math-container">$\max_{y_1, ..., y_n} \frac{1}{n} \sum_{i=1} \text{log} P(y_i | y_1, ..., y_{n-1}, x_i) \approx \frac{1}{n} \sum_{i=1} \text{log} \max_{y_i} P(y_i| y_1, ..., y_{n-1}, x_i)$</span></p>
  <p>taking maximum log probability of each step individually.</p>
  <p>In principle one could use beam search instead of just greedy. However, this only applies to models like CRFs or LSTMs where the labels of each step are not indpendent. For vanilla BERT it doesn't make a difference.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-08-25T09:43:48.827Z
Answers.LastActivityDate: 2022-08-25T09:43:48.827Z
Answers.UserDisplayName: Nether_Nova
Body: <p>I have encountered difficulties understanding the calculation of Maximum Normalized Log Probabilities acording to <a href="https://aclanthology.org/W17-2630.pdf" rel="nofollow noreferrer">Shen et al.</a>.</p>
  <p><a href="https://i.sstatic.net/jmIcm.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/jmIcm.png" alt="formular for MNLP" /></a></p>
  <p>With n being the sequence length, yi the label of word i. Xij is the representation (the input).</p>
  <p>Let me describe my setting. I'm using the implementation of BERT provided by pytorch to finetune a BERT-Base model for sequence labeling. The goal is to determine a label for each word in the sequence. There are three possible labels for each word. The basic model is set up and runs no problem. Implementing Active Learning based on Maximum Normalized Log Probability is not the problem. Im just not sure if i understand the formular correcly. The Model output is a 128 long (since each sequence is 128 words. Overflow from wordPiece Tokenization is not relevant) list of Labels.</p>
  <p>So far i'm using the softmax-function to get the probability for each label instead of the logits the BERT-Model provides as i hand it the unlabeled data. Then i calculate the log probabilities for every label via the log10 function. In the end I sum up the maxima of the predictions  over all the words (=the probability of the predicted label for each word) and average them. This is my MNLP value i use to identify the most uncertain instances.</p>
  <p>Am i doing it right? My main problem is the max with lowered yi ... yn. As far as i understand it, its used to indicate that only the maximum of the label likelihoods per word is relevant, aka. the probability of the predicted label.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-08-11T10:32:08.287Z
Comments.Text: I am actually working on something similar, BERT-based NER for medical entities. Unfortunately, I too am currently confused on the same bit, on how to incorporate MNLP for AL. Did you have any success in this regard?
Comments.UserDisplayName: Saad Khan
Tags: <neural-networks><deep-learning><natural-language-processing><definitions><bert>
Title: Hot to calculate Maximum Normalized log Probability for Active Learning with BERT
UserDisplayName: Tobias H 
ThreadText: # Hot to calculate Maximum Normalized log Probability for Active Learning with BERT
  
  (Asked by: username_0 on 2022-07-20)
  
  I have encountered difficulties understanding the calculation of Maximum Normalized Log Probabilities acording to [Shen et al.](https://aclanthology.org/W17-2630.pdf).
  
  [![formular for MNLP](https://i.sstatic.net/jmIcm.png)](https://i.sstatic.net/jmIcm.png)
  
  With n being the sequence length, yi the label of word i. Xij is the representation (the input).
  
  Let me describe my setting. I'm using the implementation of BERT provided by pytorch to finetune a BERT-Base model for sequence labeling. The goal is to determine a label for each word in the sequence. There are three possible labels for each word. The basic model is set up and runs no problem. Implementing Active Learning based on Maximum Normalized Log Probability is not the problem. Im just not sure if i understand the formular correcly. The Model output is a 128 long (since each sequence is 128 words. Overflow from wordPiece Tokenization is not relevant) list of Labels.
  
  So far i'm using the softmax-function to get the probability for each label instead of the logits the BERT-Model provides as i hand it the unlabeled data. Then i calculate the log probabilities for every label via the log10 function. In the end I sum up the maxima of the predictions over all the words (=the probability of the predicted label for each word) and average them. This is my MNLP value i use to identify the most uncertain instances.
  
  Am i doing it right? My main problem is the max with lowered yi ... yn. As far as i understand it, its used to indicate that only the maximum of the label likelihoods per word is relevant, aka. the probability of the predicted label.
  
  **Tags:** neural-networks, deep-learning, natural-language-processing, definitions, bert
  
  > **Comments:**
  > **username_1**: I am actually working on something similar, BERT-based NER for medical entities. Unfortunately, I too am currently confused on the same bit, on how to incorporate MNLP for AL. Did you have any success in this regard?  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2022-08-25. Score: 0
  
  Yes, you are doing it right.
  
  There is an implementation of MNLP here: <https://github.com/VectorInstitute/DANER/blob/2b5feadda6c1d35ab7be09277339c490396bc49e/backend/models/al_model.py#L193>
  
  It looks like it is the greedy approximation: $\max\_{y\_1, ..., y\_n} \frac{1}{n} \sum\_{i=1} \text{log} P(y\_i | y\_1, ..., y\_{n-1}, x\_i) \approx \frac{1}{n} \sum\_{i=1} \text{log} \max\_{y\_i} P(y\_i| y\_1, ..., y\_{n-1}, x\_i)$
  
  taking maximum log probability of each step individually.
  
  In principle one could use beam search instead of just greedy. However, this only applies to models like CRFs or LSTMs where the labels of each step are not indpendent. For vanilla BERT it doesn't make a difference.
  
  
  ---
  
  