title: The Impact of Hyperparameters on Large Language Model Inference
    Performance: An Evaluation of vLLM and HuggingFace Pipelines
summary: The recent surge of open-source large language models (LLMs) enables
  developers to create AI-based solutions while maintaining control over aspects
  such as privacy and compliance, thereby providing governance and ownership of
  the model deployment process. To utilize these LLMs, inference engines are
  needed. These engines load the model's weights onto available resources, such
  as GPUs, and process queries to generate responses. The speed of inference, or
  performance, of the LLM, is critical for real-time applications, as it computes
  millions or billions of floating point operations per inference. Recently,
  advanced inference engines such as vLLM have emerged, incorporating novel
  mechanisms such as efficient memory management to achieve state-of-the-art
  performance. In this paper, we analyze the performance, particularly the
  throughput (tokens generated per unit of time), of 20 LLMs using two inference
  libraries: vLLM and HuggingFace's pipelines. We investigate how various
  hyperparameters, which developers must configure, influence inference
  performance. Our results reveal that throughput landscapes are irregular, with
  distinct peaks, highlighting the importance of hyperparameter optimization to
  achieve maximum performance. We also show that applying hyperparameter
  optimization when upgrading or downgrading the GPU model used for inference can
  improve throughput from HuggingFace pipelines by an average of 9.16% and 13.7%,
  respectively.
paper_authors._id: 66b1cd82b1c62888cf9595ff
paper_authors.name: Matias Martinez
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-08-06T07:43:50.846Z
paper_authors._id: 63e24ad152b7578dba484031
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63e24ad152b7578dba484031/N1qX1rJ5cYdiOAQAULAV9.jpeg
paper_authors.fullname: Matias Martinez
paper_authors.type: user
paper_authors.user: martinezmatias
paper_title: The Impact of Hyperparameters on Large Language Model Inference
    Performance: An Evaluation of vLLM and HuggingFace Pipelines
paper_summary: The recent surge of open-source large language models (LLMs) enables
  developers to create AI-based solutions while maintaining control over aspects
  such as privacy and compliance, thereby providing governance and ownership of
  the model deployment process. To utilize these LLMs, inference engines are
  needed. These engines load the model's weights onto available resources, such
  as GPUs, and process queries to generate responses. The speed of inference, or
  performance, of the LLM, is critical for real-time applications, as it computes
  millions or billions of floating point operations per inference. Recently,
  advanced inference engines such as vLLM have emerged, incorporating novel
  mechanisms such as efficient memory management to achieve state-of-the-art
  performance. In this paper, we analyze the performance, particularly the
  throughput (tokens generated per unit of time), of 20 LLMs using two inference
  libraries: vLLM and HuggingFace's pipelines. We investigate how various
  hyperparameters, which developers must configure, influence inference
  performance. Our results reveal that throughput landscapes are irregular, with
  distinct peaks, highlighting the importance of hyperparameter optimization to
  achieve maximum performance. We also show that applying hyperparameter
  optimization when upgrading or downgrading the GPU model used for inference can
  improve throughput from HuggingFace pipelines by an average of 9.16% and 13.7%,
  respectively.
paper_ai_keywords: multimodal models large language models (LLMs) inference engines model deployment GPU inference performance floating point operations vLLM efficient memory management throughput hyperparameter optimization