Hmm yes, you should actually divide by encodings.input_ids.size(1) since i doesn’t account for the length of the last stride.
I also just spotted another bug. When the length of the last segment is less than stride, the log_likelihood calculation is slightly off. The difference in scores won’t be significant, but I’ve update the guide on master. This should be right:
max_length = model.config.n_positions
stride = 512

lls = []
for i in tqdm(range(0, encodings.input_ids.size(1), stride)):
    begin_loc = max(i + stride - max_length, 0)
    end_loc = min(i + stride, encodings.input_ids.size(1))
    trg_len = end_loc - i    # may be different from stride on last loop
    input_ids = encodings.input_ids[:,begin_loc:end_loc].to(device)
    target_ids = input_ids.clone()
    target_ids[:,:-trg_len] = -100

    with torch.no_grad():
        outputs = model(input_ids, labels=target_ids)
        log_likelihood = outputs[0] * trg_len

    lls.append(log_likelihood)

ppl = torch.exp(torch.stack(lls).sum() / end_loc)

Does that answer your question?