Scoring each word from the sentence using Pretrained LM
# ❓ Questions & Help


## Details
I am interested in scoring each word  from the sentence using the LM. Detect how likely possible is the word in the given context by scoring it using the LM.

Is there a way out for this ?

Something like [this](https://github.com/huggingface/transformers/issues/5000#issuecomment-647560847)?<|||||>You mean to say that use soft-max on the the hidden states of every word to get the score ?

I think that should work.<|||||>Great!<|||||>Since the code referenced tries to get the next prediction and I want to score existing sentence I modified to code to take the soft-max on the 2nd axis which is the word. Here is output. But I am unable to score the sentence or find the perplexity.

```
from transformers import GPT2LMHeadModel, GPT2Tokenizer
import torch

model = GPT2LMHeadModel.from_pretrained("gpt2")
tokenizer = GPT2Tokenizer.from_pretrained("gpt2")

inputs = tokenizer.encode("This is just now", return_tensors="pt")
output = model(inputs)

model_output = output[0]
last_token_softmax = torch.softmax(model_output, dim=-1).squeeze()
last_token_softmax.shape
top_n_values = last_token_softmax.topk(1)
for index, value in zip(top_n_values.indices, top_n_values.values):
    print("Score: ", value.tolist())
    print("This is just" + tokenizer.decode(index.tolist()))
```

```
Score:  [0.04588045924901962]
This is just is
Score:  [0.1578938215970993]
This is just a
Score:  [0.17527997493743896]
This is just a
Score:  [0.13700434565544128]
This is just,

```<|||||>Hi, welcome back 🤗!

We now have a document for perplexity especially: https://huggingface.co/transformers/perplexity.html

Could you take a look and let me know if it's helpful for your use-case?