Longformer Memory Consumption query
Hello,
Apologies if I am misunderstanding it, but if I use roberta with a max sequence length of 256 and I can run, for example, say, a batch size of 64 on one gpu for a task, with the longformer, can I use the same batch size with a window length of 256 with the max sequence length being 4096?
A question that might be of interest to @patrickvonplaten :)<|||||>Hey @PrudhviRaj12, 
my best answer would be to try it out :-) If can definitely work if you have enough GPU RAM. 
My best guess would be that in the scenario described by you above the Longformer version would require ca. `num_chunks` * `required_gpu_ram_for_roberta` and `num_chunks` in your case would be 4096 / 256 = 16. So you would need a lot of RAM to run `batch_size=64` and `max_length=4096` with Longformer, most likely not enough for one GPU (even if fp16).<|||||>I would also suggest adding gradient_checkpointing=True when you load your model with from_pretrained. This recent addition to the HF code base allowed me to go from using BERT with a max sequence length of 128-256 before running out of memory to now being able to use Longformer with a max seq length of up to 4096 on the same GPU setup!

This thread helped me and may also help you:
https://github.com/allenai/longformer/issues/80<|||||>Thanks @patrickvonplaten  - I misunderstood the paper then. @HugToDebug thanks for your suggestion - I tried that but I am getting this warning 

```
None of the inputs have requires_grad=True. Gradients will be None
warnings.warn("None of the inputs have requires_grad=True. Gradients will be None"
```

when calling model.forward(sequence inputs, attention masks) with any model (be it longformer or bert or roberta) and the performance of the model is completely off of the same batch experimental setting with and without gradient checkpointing. Probably I am doing something wrong, I'll check that thread. 

I am only training the last N layers of the bert/roberta for my task, and I am setting requires grad = False for all the other layers and I am getting that warning. When I remove that condition of setting requires grad = False for some layers and leaving them true for all, I am not getting that warning. Any idea how to get around that issue?<|||||>Update: I was able to get rid of that warning by making one of the embedding weight matrices trainable (in my case - Roberta, token type embedding). It was only adding 768 more trainable parameters, but I am getting OOM. I had to cut down the batch size 4x to get it running on one gpu without OOM. Not sure why adding just 768 trainable params had that of an impact.<|||||>This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
