Title: Hello hacker news,<p>I’m the maintainer of liteLLM() - package to simplify input&#x2F;output to OpenAI, Azure, Cohere, Anthropic, Hugging face API Endpoints: <a href="https:&#x2F;&#x2F;github.com&#x2F;BerriAI&#x2F;litellm&#x2F;">https:&#x2F;&#x2F;github.com&#x2F;BerriAI&#x2F;litellm&#x2F;</a><p>We’re open sourcing our implementation of liteLLM proxy: <a href="https:&#x2F;&#x2F;github.com&#x2F;BerriAI&#x2F;litellm&#x2F;blob&#x2F;main&#x2F;cookbook&#x2F;proxy-server&#x2F;readme.md">https:&#x2F;&#x2F;github.com&#x2F;BerriAI&#x2F;litellm&#x2F;blob&#x2F;main&#x2F;cookbook&#x2F;proxy-...</a><p>TLDR: It has one API endpoint &#x2F;chat&#x2F;completions and standardizes input&#x2F;output for 50+ LLM models + handles logging, error tracking, caching, streaming<p>What can liteLLM proxy do?
- It’s a central place to manage all LLM provider integrations<p>- Consistent Input&#x2F;Output Format
    - Call all models using the OpenAI format: completion(model, messages)
    - Text responses will always be available at [&#x27;choices&#x27;][0][&#x27;message&#x27;][&#x27;content&#x27;]<p>- Error Handling Using Model Fallbacks (if GPT-4 fails, try llama2)<p>- Logging - Log Requests, Responses and Errors to Supabase, Posthog, Mixpanel, Sentry, Helicone<p>- Token Usage &amp; Spend - Track Input + Completion tokens used + Spend&#x2F;model<p>- Caching - Implementation of Semantic Caching<p>- Streaming &amp; Async Support - Return generators to stream text responses<p>You can deploy liteLLM to your own infrastructure using Railway, GCP, AWS, Azure<p>Happy completion() !
Upvote: