Nicemleme, Quantization Experiment, Bitsandbytesconfig, Model Quantization, Üretken Yapay Zeka.
BitsAndBytesConfig(load_in_4bit=True, bnb_4bit_use_double_quant=True, bnb_4bit_quant_type="nf4", bnb_4bit_compute_dtype=torch.float16) model = AutoModelForCausalLM.from_pretrained(MODEL, cache_dir="./model", quantization_config=quantization_config, low_cpu_mem_usage = True) Modele isteğin atıldığı