One of the reasons, I think about, is that the attention weights and so on within transformer model are doubled when the input length is doubled. Depending on the task, it may increase it by a factor of four.