Best way to offload layers on cPU/RAM for 6x24gb GPU

by djdeniro - opened 13 days ago

13 days ago

Hello! For 256 RAM + 144 VRAM what is best way to offload layers to CPU?

For Q2_K_XL and Q4_K_XL.

Thanks!

Unsloth AI org 13 days ago

I wrote about on how to offload more layers for more performance here: https://docs.unsloth.ai/basics/qwen3-coder#improving-generation-speed :)

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment