You should finetune original R1 671B
Seriously, it's just too good in rp and uncensored. First time I felt like I was talking to Opus-level model in RP.
With unsloth dynamic quants technique, it runs on consumer hardware, they say that even 20gb of ram is enough (with llama.cpp expert offloading). And they get good perfomance on one or two rtx 3090 with 24gb vram without noticiably dumbing down the model. Also i feel like it doesn't output its thoughts in RP, just starts writing answers directly like non-reasoing models, but they are so smart anyways.
Or maybe you can create datasets with that model (finetuning is more interesting though).
Hoping for deepseek in V5 series.
Seriously, it's just too good in rp and uncensored. First time I felt like I was talking to Opus-level model in RP.
With unsloth dynamic quants technique, it runs on consumer hardware, they say that even 20gb of ram is enough (with llama.cpp expert offloading). And they get good perfomance on one or two rtx 3090 with 24gb vram without noticiably dumbing down the model. Also i feel like it doesn't output its thoughts in RP, just starts writing answers directly like non-reasoing models, but they are so smart anyways.
Or maybe you can create datasets with that model (finetuning is more interesting though).
Hoping for deepseek in V5 series.
20gb ram and one 3090? Seriously? Amazing assuming its not a typo.
20gb ram and one 3090? Seriously? Amazing assuming its not a typo.
Yes, it's not a typo and it really is amazing.
They say that it will run on 20 gb ram, but perfomance will not be optimal. But they recommend running it on one 3090 and provide chart for layer offload options.
https://unsloth.ai/blog/deepseekr1-dynamic
They also say it outputs only 1 incorrect tokken in 500 and it can be mitigated with min_p 0.05
Also I saw someone from magnum team talking about finetuning full deepseek on reddit (someone asked about finetuning distills and the answer was that distills are bad in rp, but they will consider finetuning full model).
But seriously, even without fintuning it beats every other local model I tried in 7-123b range.
I'll have to give it a try, I cant see it being great on my 24gb vram and 32gb ram machine.
we'll definitely consider it, As Deepseek has been loved by a lot of people in Anthracite. However currently we're focusing on improving our RL and SFT steps - Gotta learn to walk before you run.