Making AMD GPUs competitive for LLM inference
One of the authors here. Glad it’s on HackerNews!<p>There are two points I personally wanted to make through this project:<p>1) With a sufficiently optimized software stack, AMD GPUs can be sufficiently cost-efficient to use in LLM serving;
2) ML compilation (MLC) techniques, through its underlying TVM Unity software stack, are the best fit in terms of cross-hardware generalizable performance optimizations, quickly delivering time-to-market values, etc.<p>So far, to the best of our knowledge, MLC LLM delivers the best performance across NVIDIA and AMD GPUs in single-batch inference on quantized models, and batched/distributed inference is on the horizon too.