For those who were following this post, I tried in a more rigorous way with some (around 10) articles from the CNN/DM and the length_penalty parameter does change the output (actually a lot, from 200 to 500 tokens). However, it is still a mystery for me why sometimes it does not influence it at all.
The code I have used:
import torch
torch.manual_seed(42)

summ = {}
model.to('cuda')
for i, a in enumerate(articles):
    summ[i] = []
    for lp in [0.1, 1, 2]:
        summ[i].append(
            tokenizer.decode(
                model.generate(
                    tokenizer.encode(a, truncation=True, return_tensors='pt').to('cuda'),
                    length_penalty=lp)[0],
                skip_special_tokens=True))