Yes, yes, they are, but for 90% of the steps, they use 128 sequence lengths and for the remaining 10% 512. The reasoning is that only 10% is enough to learn positional embeddings, and using 512 for the whole training is a waste. Now because of this change, one has to lower the batch size, adjust the learning rate etc. It is a two-phase pre-training process.