@valhalla thank you very much for the advise, it worked.
I have been able to train the model with a batch size of 3. Still I need to improve it as 1 epoch will take 60 hours  .
I am thinking about distributed training on the cloud. Still I need to iterate the model.
What do you think about these trade-offs?


Shorter vocab size or bigger as it will result in shorter tokenized sequences?

Bucket longer sequences that 2000 and mean vector representations?

Less layers or smaller hidden and intermediate size?

Reducing the number of attention heads?

Which of these parameters have a higher memory? Is there a methodology to choose them?
Thank you very much for the advices.