Minimum gpu ram capacity
My laptop GPU is RTX 3070 Ti Laptop.
When I tried to run model. I got error killed. 50% of the time it progresses and then stops. What is the minimum capacity?
- If you want to run the model in 4-bit quantization it should need
6GBof GPU. - If you want to fine-tune the model in 4bit quantization you should need at least
15GBGPU. - if you want to run the full model you should need at least
16GBGPU.
I'm not sure which model you're running, I'll assume it is 3.1 8B instruct, because this is the community for that.
I haven't set it up a lap top of any kind,
but I have set it up on a windows 10 pc using a geforce GT 1030 GPU with 2 GB of GDDR, and I've set it up on Fedora server/ Fedora Workstation/Linux Mint Cinnamon/ Ubuntu with the same hardware.
It maybe too late to suggest, it's been 27 days, but before you assume you don't have enough memory --- from the specs I can find on the for your computer, it has 8 GB GDDR6,,, way more than I had -- and before you run it in 4-bit quantazation, you should try running on the cpu.
I also don't know what script you're running but whereever you can find the parameter 'device=' or 'device_mapping' change that value to 'cpu' instead of 'auto' or 'cuda', then try running the script and tell me what you see.