Answers.Body: <p>Possibly a bit late to the answer, but I doubt you'd be able to run GPT-2 774M in FP32 on 2070 Super which has 8GB VRAM. I know it's not an exact comparison, but fine-tuning BERT Large (345M) in FP32 easily takes more than 10GB of VRAM. You might be able to run GPT-2 774M if you run it in FP16.</p>
  <p>Alternatively, you can use Google Collab TPUs which provide at 11GB+ VRAM. Here's a good source listing a few posts about fine tuning GTP-2 1.5B on Google Collab TPUs: <a href="https://news.ycombinator.com/item?id=21456025" rel="nofollow noreferrer">https://news.ycombinator.com/item?id=21456025</a></p>
  <p>And here's the notebook itself demonstrating the process: <a href="https://colab.research.google.com/drive/1BXry0kcm869-RVHHiY6NZmY9uBzbkf1Q#scrollTo=lP1InuxJTD6a" rel="nofollow noreferrer">https://colab.research.google.com/drive/1BXry0kcm869-RVHHiY6NZmY9uBzbkf1Q#scrollTo=lP1InuxJTD6a</a></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-04-24T11:35:19.637Z
Answers.LastActivityDate: 2021-04-24T11:35:19.637Z
Answers.UserDisplayName: Dan Pavlov
Body: <p>I wonder if there's anyone who has actually succeeded in fine-tuning GPT-2's 774M model without using cloud TPU's. My GeForce RTX 2070 SUPER couldn't handle it in previous attempts.</p>
  <p>I'm running TensorFlow 1.14.0 with CUDA V 9.1 on Ubuntu 18.04. For fine-tuning I'm using <a href="https://github.com/minimaxir/gpt-2-simple" rel="nofollow noreferrer">gpt-2-simple</a>.</p>
  <p>When fine-tuning using the 77M model, I keep running into OOM errors, such as:
  <code>W tensorflow/core/common_runtime/bfc_allocator.cc:314] Allocator (GPU_0_bfc) ran out of memory trying to allocate 6.25MiB (rounded to 6553600).  Current allocation summary follows.</code></p>
  <p>So far I've tried:</p>
  <ul>
  <li>Using different a optimizer (<code>RMSPropOptimizer</code> instead of <code>AdamOptimizer</code>)</li>
  <li>Setting batch-size to 1</li>
  <li><code>use_memory_saving_gradients</code></li>
  <li><code>only_train_transformer_layers</code></li>
  </ul>
  <p>Fine-tuning works smoothly on the 355M model.</p>
  <p>So what I'm really asking is:</p>
  <ul>
  <li>is it possible to fine-tune GPT-2's 774M model without industrial-sized hardware?</li>
  <li>if so, please tell me about your successful attempts</li>
  <li>apart from hardware-recommendations, how could fine-tuning be optimized to make 77M fit in memory?</li>
  </ul>
  
Tags: <natural-language-processing><gpt><fine-tuning><gpt-2>
Title: GPT-2: (Hardware) requirements for fine-tuning the 774M model
UserDisplayName: Comfort Eagle
ThreadText: # GPT-2: (Hardware) requirements for fine-tuning the 774M model
  
  (Asked by: username_0 on 2020-04-24)
  
  I wonder if there's anyone who has actually succeeded in fine-tuning GPT-2's 774M model without using cloud TPU's. My GeForce RTX 2070 SUPER couldn't handle it in previous attempts.
  
  I'm running TensorFlow 1.14.0 with CUDA V 9.1 on Ubuntu 18.04. For fine-tuning I'm using [gpt-2-simple](https://github.com/minimaxir/gpt-2-simple).
  
  When fine-tuning using the 77M model, I keep running into OOM errors, such as:
  `W tensorflow/core/common_runtime/bfc_allocator.cc:314] Allocator (GPU_0_bfc) ran out of memory trying to allocate 6.25MiB (rounded to 6553600). Current allocation summary follows.`
  
  So far I've tried:
  
  * Using different a optimizer (`RMSPropOptimizer` instead of `AdamOptimizer`)
  * Setting batch-size to 1
  * `use_memory_saving_gradients`
  * `only_train_transformer_layers`
  
  Fine-tuning works smoothly on the 355M model.
  
  So what I'm really asking is:
  
  * is it possible to fine-tune GPT-2's 774M model without industrial-sized hardware?
  * if so, please tell me about your successful attempts
  * apart from hardware-recommendations, how could fine-tuning be optimized to make 77M fit in memory?
  
  **Tags:** natural-language-processing, gpt, fine-tuning, gpt-2
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2021-04-24. Score: 1
  
  Possibly a bit late to the answer, but I doubt you'd be able to run GPT-2 774M in FP32 on 2070 Super which has 8GB VRAM. I know it's not an exact comparison, but fine-tuning BERT Large (345M) in FP32 easily takes more than 10GB of VRAM. You might be able to run GPT-2 774M if you run it in FP16.
  
  Alternatively, you can use Google Collab TPUs which provide at 11GB+ VRAM. Here's a good source listing a few posts about fine tuning GTP-2 1.5B on Google Collab TPUs: <https://news.ycombinator.com/item?id=21456025>
  
  And here's the notebook itself demonstrating the process: <https://colab.research.google.com/drive/1BXry0kcm869-RVHHiY6NZmY9uBzbkf1Q#scrollTo=lP1InuxJTD6a>
  
  
  ---
  
  