Getting Sentence level log probabilities using this model
So I was trying to implement the SLOR score, using the transformer-XL, mostly avoiding training. But given the XL model and the sentence, how could i go about getting the sentence level log probability?


Attached is the formula I'm trying to implement.
![image](https://user-images.githubusercontent.com/18056781/56021217-81d6f700-5d26-11e9-8c46-9b88f0277398.png)


Thanks a lot for your help.

So p_M(S) is just the output of the model right?

For p_u(S), I think the easiest is probably to use the empirical probabilities.
`TransfoXLTokenizer` has a counter to store words frequencies [here](https://github.com/huggingface/pytorch-pretrained-BERT/blob/master/pytorch_pretrained_bert/tokenization_transfo_xl.py#L98) which should be populated in the "pretrained" tokenizer so I would use and normalize this to get unconditional probabilities for each word and then compute SLOR.<|||||>@thomwolf So the code would look something like : 
```sentence = "Where is the dog going"
s_ids = convert_sentence_to_ids(sentence)
config = TransfoXLConfig()
model = TransfoXLModel(config)
last_hidden_state, new_mems = model(input_ids)
for i in sentence : 
     uni_prob =  freq(token)/no_of_tokens
     sent_uni_prob * = uni_prob 
SLOR = (1/len(sentence))* last_hidden_state - sent_uni_prob 
```

or am i mistaken about the idea that the last_hidden_state is the sentence probability i'm looking for<|||||>This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
<|||||>In case someone found this, I found an implementation that seems legit here https://github.com/sgondala/GoogleConceptualCaptioning/commit/981e1b61ca5f84052f6237402319714d7fe70b80.