content.raw: Just tried LitServe from the good folks at 
content.type: text
content.value: Just tried LitServe from the good folks at 
content.raw: @LightningAI
content.type: mention
content.user: LightningAI
content.raw: !
content.type: text
content.value: !
content.type: new_line
content.type: new_line
content.raw: Between llama.cpp and vLLM, there is a small gap where a few large models are not deployable!
content.type: text
content.value: Between llama.cpp and vLLM, there is a small gap where a few large models are not deployable!
content.type: new_line
content.type: new_line
content.raw: That's where LitServe comes in!
content.type: text
content.value: That's where LitServe comes in!
content.type: new_line
content.type: new_line
content.raw: LitServe is a high-throughput serving engine for AI models built on FastAPI.
content.type: text
content.value: LitServe is a high-throughput serving engine for AI models built on FastAPI.
content.type: new_line
content.type: new_line
content.raw: Yes, built on FastAPI. That's where the advantage and the issue lie.
content.type: text
content.value: Yes, built on FastAPI. That's where the advantage and the issue lie.
content.type: new_line
content.type: new_line
content.raw: It's extremely flexible and supports multi-modality and a variety of models out of the box.
content.type: text
content.value: It's extremely flexible and supports multi-modality and a variety of models out of the box.
content.type: new_line
content.type: new_line
content.raw: But in my testing, it lags far behind in speed compared to vLLM.
content.type: text
content.value: But in my testing, it lags far behind in speed compared to vLLM.
content.type: new_line
content.type: new_line
content.raw: Also, no OpenAI API-compatible endpoint is available as of now.
content.type: text
content.value: Also, no OpenAI API-compatible endpoint is available as of now.
content.type: new_line
content.type: new_line
content.raw: But as we move to multi-modal models and agents, this serves as a good starting point. However, it’s got to become faster...
content.type: text
content.value: But as we move to multi-modal models and agents, this serves as a good starting point. However, it’s got to become faster...
content.type: new_line
content.type: new_line
content.raw: GitHub: 
content.type: text
content.value: GitHub: 
content.href: https://github.com/Lightning-AI/LitServe
content.raw: https://github.com/Lightning-AI/LitServe
content.type: link
rawContent: Just tried LitServe from the good folks at @LightningAI!
  
  Between llama.cpp and vLLM, there is a small gap where a few large models are not deployable!
  
  That's where LitServe comes in!
  
  LitServe is a high-throughput serving engine for AI models built on FastAPI.
  
  Yes, built on FastAPI. That's where the advantage and the issue lie.
  
  It's extremely flexible and supports multi-modality and a variety of models out of the box.
  
  But in my testing, it lags far behind in speed compared to vLLM.
  
  Also, no OpenAI API-compatible endpoint is available as of now.
  
  But as we move to multi-modal models and agents, this serves as a good starting point. However, it’s got to become faster...
  
  GitHub: https://github.com/Lightning-AI/LitServe
reactions.reaction: 👀
reactions.users: John6666 victor theQuert
reactions.reaction: 🚀
reactions.users: Norod78 aniketmaurya
commentators._id: 63045a887373aacccd8ad312
commentators.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63045a887373aacccd8ad312/pWrzmd6J4280IhHaBdVeu.jpeg
commentators.fullname: Aniket Maurya
commentators.name: aniketmaurya
commentators.type: user