@Zhylkaaa  That’s a good question, I don’t know the answer fully. There are a lot of discrepancies between the paper and the fairseq code. You could try to use the linked
command 4 and see how big you can batch with that. If it’s different, you can ask on fairseq.
Otherwise,  could you just do grad_acc=32?




why there are 1024 pos_embeddings, when paper authors write about pre-training 512? are they randomly initialised or is it something different?

This command 4 has --max_tokens=1024, 128 or 64 work better in my experience.
The state dict for mbart had 1024 trained positional embeddings, so we ported all of them.