<h3 dir="auto">System Info</h3>
<ul dir="auto">
<li><code class="notranslate">transformers</code> version: 4.22.1</li>
<li>Platform: Linux-4.19.0-21-cloud-amd64-x86_64-with-glibc2.28</li>
<li>Python version: 3.9.5</li>
</ul>
<h3 dir="auto">Who can help?</h3>
<p dir="auto"><a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/Rocketknight1/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/Rocketknight1">@Rocketknight1</a></p>
<h3 dir="auto">Information</h3>
<ul class="contains-task-list">
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> The official example scripts</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> My own modified scripts</li>
</ul>
<h3 dir="auto">Tasks</h3>
<ul class="contains-task-list">
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> An officially supported task in the <code class="notranslate">examples</code> folder (such as GLUE/SQuAD, ...)</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> My own task or dataset (give details below)</li>
</ul>
<h3 dir="auto">Reproduction</h3>
<p dir="auto">Using tf.keras.Model version of gpt2, and training as casual language modelling.</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="gpt2_model.compile(..., metrics=[PerplexityMetric()]) #using inner loss (hf_compute_loss)
gpt2_model.fit(...)"><pre class="notranslate"><code class="notranslate">gpt2_model.compile(..., metrics=[PerplexityMetric()]) #using inner loss (hf_compute_loss)
gpt2_model.fit(...)
</code></pre></div>
<p dir="auto">The problem occurs when metric is calculated. I did not come up with an idea to use inner loss of transformers in tf.keras.metric.Metric (if anyone knows how it's good to rid of duplicate calculation) . So calculating loss as written in def hf_compute_loss in TFCausalLanguageModelingLoss. PerplexityMetric as:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="class PerplexityMetric(tf.keras.metrics.Metric):    
    def __init__(self, name='perplexity', **kwargs):
      super(PerplexityMetric, self).__init__(name=name, **kwargs)
      self.cross_entropy = tf.keras.losses.SparseCategoricalCrossentropy(
            from_logits=True, reduction=tf.keras.losses.Reduction.NONE
        )
      self.perplexity = self.add_weight(name='tp', initializer='zeros')
	
    def _calculate_perplexity(self, real, pred):	       	 			
        unmasked_loss = self.cross_entropy(tf.nn.relu(real), pred)
        # make sure only labels that are not equal to -100 affect the loss
        loss_mask = tf.cast(real != -100, dtype=unmasked_loss.dtype)
        masked_loss = unmasked_loss * loss_mask
        reduced_masked_loss = tf.reduce_sum(masked_loss) / tf.reduce_sum(
            loss_mask
        )
        loss_ = tf.reshape(reduced_masked_loss, (1,))        
        perplexity = tf.math.exp(loss_[-1])        
        return perplexity 

    def update_state(self, y_true, y_pred, sample_weight=None):           
        perplexity = self._calculate_perplexity(y_true, y_pred)	        	
        self.perplexity.assign(perplexity)
        
    def result(self):
      return self.perplexity

    def reset_state(self):
      # reset at the start of each epoch.
      self.perplexity.assign(0.0)"><pre class="notranslate"><code class="notranslate">class PerplexityMetric(tf.keras.metrics.Metric):    
    def __init__(self, name='perplexity', **kwargs):
      super(PerplexityMetric, self).__init__(name=name, **kwargs)
      self.cross_entropy = tf.keras.losses.SparseCategoricalCrossentropy(
            from_logits=True, reduction=tf.keras.losses.Reduction.NONE
        )
      self.perplexity = self.add_weight(name='tp', initializer='zeros')
	
    def _calculate_perplexity(self, real, pred):	       	 			
        unmasked_loss = self.cross_entropy(tf.nn.relu(real), pred)
        # make sure only labels that are not equal to -100 affect the loss
        loss_mask = tf.cast(real != -100, dtype=unmasked_loss.dtype)
        masked_loss = unmasked_loss * loss_mask
        reduced_masked_loss = tf.reduce_sum(masked_loss) / tf.reduce_sum(
            loss_mask
        )
        loss_ = tf.reshape(reduced_masked_loss, (1,))        
        perplexity = tf.math.exp(loss_[-1])        
        return perplexity 

    def update_state(self, y_true, y_pred, sample_weight=None):           
        perplexity = self._calculate_perplexity(y_true, y_pred)	        	
        self.perplexity.assign(perplexity)
        
    def result(self):
      return self.perplexity

    def reset_state(self):
      # reset at the start of each epoch.
      self.perplexity.assign(0.0)
</code></pre></div>
<h3 dir="auto">Expected behavior</h3>
<p dir="auto">Both inner loss calculation and the loss calculation in the metric are same. What I expect is that calculated losses are the same in training output. Last float within brackets is loss calculated in PerplexityMetric() (<em>loss</em>_).</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="1/7 [===&gt;..........................] - ETA: 2s - loss: 8.0234 - perplexity: 1708.3318[7.44036245]
2/7 [=======&gt;......................] - ETA: 2s - loss: 8.1277 - perplexity: 1703.3676[7.78889942]
3/7 [===========&gt;..................] - ETA: 1s - loss: 8.2354 - perplexity: 2413.6597[7.75492764]
4/7 [================&gt;.............] - ETA: 1s - loss: 8.3068 - perplexity: 2333.0405[7.91714859]
5/7 [====================&gt;.........] - ETA: 0s - loss: 8.3837 - perplexity: 2743.9358[7.48942709]
6/7 [========================&gt;.....] - ETA: 0s - loss: 8.3680 - perplexity: 1789.0269[7.8069911]"><pre class="notranslate"><code class="notranslate">1/7 [===&gt;..........................] - ETA: 2s - loss: 8.0234 - perplexity: 1708.3318[7.44036245]
2/7 [=======&gt;......................] - ETA: 2s - loss: 8.1277 - perplexity: 1703.3676[7.78889942]
3/7 [===========&gt;..................] - ETA: 1s - loss: 8.2354 - perplexity: 2413.6597[7.75492764]
4/7 [================&gt;.............] - ETA: 1s - loss: 8.3068 - perplexity: 2333.0405[7.91714859]
5/7 [====================&gt;.........] - ETA: 0s - loss: 8.3837 - perplexity: 2743.9358[7.48942709]
6/7 [========================&gt;.....] - ETA: 0s - loss: 8.3680 - perplexity: 1789.0269[7.8069911]
</code></pre></div>
<p dir="auto">I did not understand the difference between loss values. Is there any wrong calculation in PerplexityMetric or is this a buggy behaviour?</p>
<p dir="auto">GPU:v100</p>
<p dir="auto">run run_glue.py with the command in the README:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="python ./examples/run_glue.py \
    --model_type bert \
    --model_name_or_path bert-base-uncased \
    --task_name $TASK_NAME \
    --do_train \
    --do_eval \
    --do_lower_case \
    --data_dir $GLUE_DIR/$TASK_NAME \
    --max_seq_length 128 \
    --per_gpu_eval_batch_size=8   \
    --per_gpu_train_batch_size=8   \
    --learning_rate 2e-5 \
    --num_train_epochs 3.0 \
    --output_dir /tmp/$TASK_NAME/"><pre class="notranslate"><code class="notranslate">python ./examples/run_glue.py \
    --model_type bert \
    --model_name_or_path bert-base-uncased \
    --task_name $TASK_NAME \
    --do_train \
    --do_eval \
    --do_lower_case \
    --data_dir $GLUE_DIR/$TASK_NAME \
    --max_seq_length 128 \
    --per_gpu_eval_batch_size=8   \
    --per_gpu_train_batch_size=8   \
    --learning_rate 2e-5 \
    --num_train_epochs 3.0 \
    --output_dir /tmp/$TASK_NAME/
</code></pre></div>
<p dir="auto">and i also run with --fp16,  but spend the same time and gpu memory, why fp16 does not work</p>