Changing LM loss function
Hi all,

I am modifying gpt2 loss function. My new code looks like that:


        lm_logits = self.lm_head(hidden_states)

        outputs = (lm_logits,) + transformer_outputs[1:]
        if labels is not None:
            # Shift so that tokens < n predict n
            shift_logits = lm_logits[..., :-1, :].contiguous()
            shift_labels = labels[..., 1:].contiguous()
            all_logits = shift_logits[0].cpu().data.numpy()
            tokenizer = GPT2Tokenizer.from_pretrained("gpt2")
            sampled_words = []
            for elem in all_logits:
                logits = elem.reshape(-1)
                exps = np.exp(numbers - np.max(logits))
                output_logits_normalized = exps / np.sum(exps)
                sampled_word = np.array(np.argmax(output_logits_normalized)).reshape([1,1])
                sampled_words.append(sampled_word)
            text = tokenizer.decode(np.array(sampled_words).reshape(-1))

            # Flatten the tokens
            loss_fct = CrossEntropyLoss(ignore_index=-1)
            loss = loss_fct(shift_logits.view(-1, shift_logits.size(-1)),
                            shift_labels.view(-1))
            print("CE Loss:", loss.cpu().data.numpy())
            l = Lyrics(text=text, language='en-us', lookback=15)
            rl = l.get_avg_rhyme_length()
            beta = 1
            rl_loss = rl * beta
            print("RL loss: ", rl_loss)
            total_loss = loss * 1/rl_loss
            print("Total loss: ", total_loss.cpu().data.numpy())
            outputs = (total_loss,) + outputs 

        return outputs  # (loss), lm_logits, presents, (all hidden_states), (attentions)


But after evaluation, every model checkpoint returns the same loss on the test set, so it seems that parameters are never updated. Could you please tell me why and how I could solve this?

Thank you a lot.
This is quite a general question. Perhaps it's more useful to put this on Stack Overflow. <|||||>We plan to have a forum associated to the repo to discuss these types of general questions.
In the meantime, we are still happy to welcome them in the PR but the visibility is limited indeed.<|||||>This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
