Fine-tuning GPT-J 6B with 16Gb of VRAM
Hi

I am trying to fine-tune GPT-J 6B with 16Gb of VRAM. Do you have any ideas/pointers on what can be done to lower the amount of VRAM needed for fine-tuning?

@StellaAthena @EricHallahan 

I've tried DeepSpeed with Zero-3 and Zero-Offload, and the model still doesn't fit. <|||||>Try checking out the instructions [here](https://github.com/kingoflolz/mesh-transformer-jax/blob/master/howto_finetune.md)<|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.