Run Llama 13B with a 6GB graphics card
On my system, using `-ngl 22` (running 22 layers on the GPU) cuts wall clock time by ~60%.<p>My system:<p>GPU: NVidia RTX 2070S (8GB VRAM)<p>CPU: AMD Ryzen 5 3600 (16GB VRAM)<p>Here&#x27;s the performance difference I see:<p>CPU only (.&#x2F;main -t 12)<p><pre><code>  llama_print_timings:        load time = 15459.43 ms
  llama_print_timings:      sample time =    23.64 ms &#x2F;    38 runs   (    0.62 ms per token)
  llama_print_timings: prompt eval time =  9338.10 ms &#x2F;   356 tokens (   26.23 ms per token)
  llama_print_timings:        eval time = 31700.73 ms &#x2F;    37 runs   (  856.78 ms per token)
  llama_print_timings:       total time = 47192.68 ms

</code></pre>
GPU (.&#x2F;main -t 12 -ngl 22)<p><pre><code>  llama_print_timings:        load time = 10285.15 ms
  llama_print_timings:      sample time =    21.60 ms &#x2F;    35 runs   (    0.62 ms per token)
  llama_print_timings: prompt eval time =  3889.65 ms &#x2F;   356 tokens (   10.93 ms per token)
  llama_print_timings:        eval time =  8126.90 ms &#x2F;    34 runs   (  239.03 ms per token)
  llama_print_timings:       total time = 18441.22 ms</code></pre>