Machine Learning, Software Development, Artificial Intelligence, Technology, Data Science.
minimum values as separate arrays. This approach lets them run inference on LLaMA models up to 2.6x faster than llama.cpp. Using smaller quantization groups improves perplexity but worsens throughput, as you’d expect. Eyeballing these plots, they also seem to have reproduced the result that 4b