Show HN: Finetune LLaMA-7B on commodity GPUs using your own text
Looks like you need about 120 GB to fine-tune the 65B model with this code at a sequence length of 512. How does the memory usage scale as the sequence length grows?