I faced the same problem. As of now, I used a 32 GB GPU (p3dn.24xlarge ec2 instance). Also reduced the number of tokens and the batch size. At present Longformer doesn’t support multiple GPUs. We can shard the training data and train it iteratively by saving the model (still exploring this). We can explore checkpoint feature in pytorch as well (https://pytorch.org/tutorials/recipes/recipes/saving_and_loading_a_general_checkpoint.html 5). Let me know if you have already found an efficient way to do this.