Skip to main contentü¶úÔ∏èüîó LangChainDocsUse casesIntegrationsAPICommunityChat our docsLangSmithJS/TS DocsSearchCTRLKGet startedIntroductionInstallationQuickstartLangChain Expression LanguageInterfaceHow toBind runtime argsConfigurable AlternativesAdd fallbacksRun arbitrary functionsUse RunnableParallel/RunnableMapRoute between multiple RunnablesCookbookLangChain Expression Language (LCEL)Why use LCEL?ModulesModel I/‚ÄãORetrievalChainsMemoryAgentsCallbacksModulesSecurityGuidesMoreLangChain Expression LanguageHow toAdd fallbacksOn this pageAdd fallbacksThere are many possible points of failure in an LLM application, whether that be issues with LLM API's, poor model outputs, issues with other integrations, etc. Fallbacks help you gracefully handle and isolate these issues.Crucially, fallbacks can be applied not only on the LLM level but on the whole runnable level.Handling LLM API Errors‚ÄãThis is maybe the most common use case for fallbacks. A request to an LLM API can fail for a variety of reasons - the API could be down, you could have hit rate limits, any number of things. Therefore, using fallbacks can help protect against these types of things.IMPORTANT: By default, a lot of the LLM wrappers catch errors and retry. You will most likely want to turn those off when working with fallbacks. Otherwise the first wrapper will keep on retrying and not failing.from langchain.chat_models import ChatOpenAI, ChatAnthropicFirst, let's mock out what happens if we hit a RateLimitError from OpenAIfrom unittest.mock import patchfrom openai.error import RateLimitError# Note that we set max_retries = 0 to avoid retrying on RateLimits, etcopenai_llm = ChatOpenAI(max_retries=0)anthropic_llm = ChatAnthropic()llm = openai_llm.with_fallbacks([anthropic_llm])# Let's use just the OpenAI LLm first, to show that we run into an errorwith patch('openai.ChatCompletion.create', side_effect=RateLimitError()):    try:         print(openai_llm.invoke("Why did the chicken cross the road?"))
There are many possible points of failure in an LLM application, whether that be issues with LLM API's, poor model outputs, issues with other integrations, etc. Fallbacks help you gracefully handle and isolate these issues.
There are many possible points of failure in an LLM application, whether that be issues with LLM API's, poor model outputs, issues with other integrations, etc. Fallbacks help you gracefully handle and isolate these issues. ->: Skip to main contentü¶úÔ∏èüîó LangChainDocsUse casesIntegrationsAPICommunityChat our docsLangSmithJS/TS DocsSearchCTRLKGet startedIntroductionInstallationQuickstartLangChain Expression LanguageInterfaceHow toBind runtime argsConfigurable AlternativesAdd fallbacksRun arbitrary functionsUse RunnableParallel/RunnableMapRoute between multiple RunnablesCookbookLangChain Expression Language (LCEL)Why use LCEL?ModulesModel I/‚ÄãORetrievalChainsMemoryAgentsCallbacksModulesSecurityGuidesMoreLangChain Expression LanguageHow toAdd fallbacksOn this pageAdd fallbacksThere are many possible points of failure in an LLM application, whether that be issues with LLM API's, poor model outputs, issues with other integrations, etc. Fallbacks help you gracefully handle and isolate these issues.Crucially, fallbacks can be applied not only on the LLM level but on the whole runnable level.Handling LLM API Errors‚ÄãThis is maybe the most common use case for fallbacks. A request to an LLM API can fail for a variety of reasons - the API could be down, you could have hit rate limits, any number of things. Therefore, using fallbacks can help protect against these types of things.IMPORTANT: By default, a lot of the LLM wrappers catch errors and retry. You will most likely want to turn those off when working with fallbacks. Otherwise the first wrapper will keep on retrying and not failing.from langchain.chat_models import ChatOpenAI, ChatAnthropicFirst, let's mock out what happens if we hit a RateLimitError from OpenAIfrom unittest.mock import patchfrom openai.error import RateLimitError# Note that we set max_retries = 0 to avoid retrying on RateLimits, etcopenai_llm = ChatOpenAI(max_retries=0)anthropic_llm = ChatAnthropic()llm = openai_llm.with_fallbacks([anthropic_llm])# Let's use just the OpenAI LLm first, to show that we run into an errorwith patch('openai.ChatCompletion.create', side_effect=RateLimitError()):    try:         print(openai_llm.invoke("Why did the chicken cross the road?"))