Just tried LitServe from the good folks at 
text
Just tried LitServe from the good folks at 
@LightningAI
mention
LightningAI
!
text
!
new_line
new_line
Between llama.cpp and vLLM, there is a small gap where a few large models are not deployable!
text
Between llama.cpp and vLLM, there is a small gap where a few large models are not deployable!
new_line
new_line
That's where LitServe comes in!
text
That's where LitServe comes in!
new_line
new_line
LitServe is a high-throughput serving engine for AI models built on FastAPI.
text
LitServe is a high-throughput serving engine for AI models built on FastAPI.
new_line
new_line
Yes, built on FastAPI. That's where the advantage and the issue lie.
text
Yes, built on FastAPI. That's where the advantage and the issue lie.
new_line
new_line
It's extremely flexible and supports multi-modality and a variety of models out of the box.
text
It's extremely flexible and supports multi-modality and a variety of models out of the box.
new_line
new_line
But in my testing, it lags far behind in speed compared to vLLM.
text
But in my testing, it lags far behind in speed compared to vLLM.
new_line
new_line
Also, no OpenAI API-compatible endpoint is available as of now.
text
Also, no OpenAI API-compatible endpoint is available as of now.
new_line
new_line
But as we move to multi-modal models and agents, this serves as a good starting point. However, it’s got to become faster...
text
But as we move to multi-modal models and agents, this serves as a good starting point. However, it’s got to become faster...
new_line
new_line
GitHub: 
text
GitHub: 
https://github.com/Lightning-AI/LitServe
https://github.com/Lightning-AI/LitServe
link
Just tried LitServe from the good folks at @LightningAI!

Between llama.cpp and vLLM, there is a small gap where a few large models are not deployable!

That's where LitServe comes in!

LitServe is a high-throughput serving engine for AI models built on FastAPI.

Yes, built on FastAPI. That's where the advantage and the issue lie.

It's extremely flexible and supports multi-modality and a variety of models out of the box.

But in my testing, it lags far behind in speed compared to vLLM.

Also, no OpenAI API-compatible endpoint is available as of now.

But as we move to multi-modal models and agents, this serves as a good starting point. However, it’s got to become faster...

GitHub: https://github.com/Lightning-AI/LitServe
👀
John6666 victor theQuert
🚀
Norod78 aniketmaurya
63045a887373aacccd8ad312
https://cdn-avatars.huggingface.co/v1/production/uploads/63045a887373aacccd8ad312/pWrzmd6J4280IhHaBdVeu.jpeg
Aniket Maurya
aniketmaurya
user