TheBloke

/

Llama-2-70B-Chat-GPTQ

like 260

Yeah I'm afraid that's not going to work. AutoGPTQ can split the model weights across multiple GPUs (though I never do that myself, and don't recommend it unless you're really short on CPU RAM), but it can't split the VRAM required for quantisation. That always goes on GPU0, and a 2080Ti is not big enough for 70B. You'll need a GPU with around 30GB VRAM, meaning you need an A100 40GB or a 48GB card like an A6000 or L40.

Out of interest, why did you want to quantise it yourself? Are you doing a set of parameters I've not done, or a different dataset?

because I want to use a dataset contains some Chinese data.
both autogptq or gptq for llama use pure english dataset for quantise by default.
This might cause the model lose more precise in Chinese, I guess?

the max vram cost is not stable while for llama1 65b, the cost is stable.
Still in progress, hope it can be done successfully...

This is the script I use for quantizing - it uses the wikitext or c4 datasets: https://gist.github.com/TheBloke/b47c50a70dd4fe653f64a12928286682#file-quant_autogptq-py

Yes I agree for Chinese language, it would be better to use a Chinese dataset.

I just re-read your first message. You say you have 22GB on the 2080Ti? The 2080Ti only has 11GB - have you modded it?

Maybe 22GB would be enough, I don't know. I used to be able to quantise 65B on 1 x 24GB GPU. But 70B is a bit bigger, and has a max sequence length of 4096.

I have never had success getting AutoGPTQ to load the model across multiple GPUs. You must not load any model weights on GPU0 else it will definitely OOM. So that would be:
max_memory = { 0: '0Gib', 1: '22GiB', 2: '22GiB', 3: '22GiB', 4: '22GiB', 5: '22GiB', 6: '22GiB', 7: '22GiB', 'cpu': '200GiB' }

But when I have tried that config before, I got a CUDA error about GPU0 not being initialised.

The only setup I have had success with is the default, where the model is loaded 100% into RAM, and then GPU0 is used automatically for quantising it. For 70B this will require about 165GB RAM.

I am quantizing 70B right now on a 48GB card, and with seqlen = 4096 it is using up to 34GB VRAM:

If you use seqlen=2048 it will be a bit less, and you can also save a little VRAM by specifying cache_examples_on_gpu=False in .quantize(). But I am not confident you are going to be able to do this in 22GB.

I suggest you rent a GPU. Runpod have L40 48GB systems with 250GB RAM for $1.14/hr.

I am quantizing 70B right now on a 48GB card, and with seqlen = 4096 it is using up to 34GB VRAM:

If you use seqlen=2048 it will be a bit less, and you can also save a little VRAM by specifying cache_examples_on_gpu=False in .quantize(). But I am not confident you are going to be able to do this in 22GB.

I suggest you rent a GPU. Runpod have L40 48GB systems with 250GB RAM for $1.14/hr.

much thanks！
I successfully quantize the model with 2048 in my machine. But seems impossible for 4096...hhhh, for renting GPU, would it cost a long time in to downloading the modle?

May I ask another question that...
Can you lora the 4bit model now? I can train and save the lora file with fintune.py in alpaca_lora_4bit.
But when I loaded the saved lora file and do inference, the output is broken within alpaca_lora_4bit and throw error within exllama

Good to hear!

No it shouldn't take a huge amount of time to download the model. It depends on the exact server of course, some have 1Gbit/s, some have 10Gbit/s. But even if it's only 1Gbit/s, to download Llama 2 130GB should only take 20-30 minutes. And then when you've made the quantisation you can upload it to Hugging Face Hub and that will be much quicker because the quantisation will be much smaller, only around 35GB.

I don't know how fast your internet is, but if you downloaded Llama 2 at 130GB presumably you can also download 35GB no problem.

yeah, I am planning to do it in colab.

Any idea with lora?