TP-Aware Dequantization
<|reference_start|>TP-Aware Dequantization: In this paper, we present a novel method that reduces model inference latency during distributed deployment of Large Language Models (LLMs). Our contribution is an optimized inference deployment scheme that address the current limitations of state-of-the-art quantization kernels when used in conjunction with Tensor Parallel (TP). Our method preserves data locality in GPU memory access patterns and exploits a priori knowledge of TP to reduce global communication. We demonstrate an up to 1.81x speedup over existing methods for Llama-70B and up to 1.78x speedup for IBM WatsonX's Granite-20B MLP layer problem sizes on A100 and H100 NVIDIA DGX Systems for a variety of TP settings.<|reference_end|>
@article{hoque2024tp-aware,
    title={TP-Aware Dequantization},
    author={Adnan Hoque, Mudhakar Srivatsa, Chih-Chieh Yang, Raghu Ganti},
    journal={arXiv preprint arXiv:2402.04925},
    year={2024},
    archivePrefix={arXiv},
    eprint={2402.04925},
    primaryClass={cs.DC cs.LG}
}