Note that we set max_retries = 0 to avoid retrying on RateLimits, etcopenai_llm = ChatOpenAI(max_retries=0)anthropic_llm = ChatAnthropic()llm = openai_llm.with_fallbacks([anthropic_llm])# Let's use just the OpenAI LLm first, to show that we run into an errorwith patch('openai.ChatCompletion.create', side_effect=RateLimitError()):    try:         print(openai_llm.invoke("Why did the chicken cross the road?"))    except:        print("Hit error")    Hit error# Now let's try with fallbacks to Anthropicwith patch('openai.ChatCompletion.create', side_effect=RateLimitError()):    try:         print(llm.invoke("Why did the the chicken cross the road?"))    except:        print("Hit error")    content=' I don\'t actually know why the chicken crossed the road, but here are some possible humorous answers:\n\n- To get to the other side!\n\n- It was too chicken to just stand there. \n\n- It wanted a change of scenery.\n\n- It wanted to show the possum it could be done.\n\n- It was on its way to a poultry farmers\' convention.\n\nThe joke plays on the double meaning of "the other side" - literally crossing the road to the other side, or the "other side" meaning the afterlife. So it\'s an anti-joke, with a silly or unexpected pun as the answer.' additional_kwargs={} example=FalseWe can use our "LLM with Fallbacks" as we would a normal LLM.from langchain.prompts import ChatPromptTemplateprompt = ChatPromptTemplate.from_messages(    [        ("system", "You're a nice assistant who always includes a compliment in your response"),        ("human", "Why did the {animal} cross the road"),    ])chain = prompt | llmwith patch('openai.ChatCompletion.create', side_effect=RateLimitError()):    try:         print(chain.invoke({"animal": "kangaroo"}))    except:        print("Hit error")    content=" I don't actually know why the kangaroo crossed the road, but I can take a guess! Here are some possible reasons:\n\n- To get to the other side (the classic joke answer!)\n\n- It was trying to
When working with language models, you may often encounter issues from the underlying APIs, whether these be rate limiting or downtime. Therefore, as you go to move your LLM applications into production it becomes more and more important to safeguard against these. That's why we've introduced the concept of fallbacks.
When working with language models, you may often encounter issues from the underlying APIs, whether these be rate limiting or downtime. Therefore, as you go to move your LLM applications into production it becomes more and more important to safeguard against these. That's why we've introduced the concept of fallbacks. ->: Note that we set max_retries = 0 to avoid retrying on RateLimits, etcopenai_llm = ChatOpenAI(max_retries=0)anthropic_llm = ChatAnthropic()llm = openai_llm.with_fallbacks([anthropic_llm])# Let's use just the OpenAI LLm first, to show that we run into an errorwith patch('openai.ChatCompletion.create', side_effect=RateLimitError()):    try:         print(openai_llm.invoke("Why did the chicken cross the road?"))    except:        print("Hit error")    Hit error# Now let's try with fallbacks to Anthropicwith patch('openai.ChatCompletion.create', side_effect=RateLimitError()):    try:         print(llm.invoke("Why did the the chicken cross the road?"))    except:        print("Hit error")    content=' I don\'t actually know why the chicken crossed the road, but here are some possible humorous answers:\n\n- To get to the other side!\n\n- It was too chicken to just stand there. \n\n- It wanted a change of scenery.\n\n- It wanted to show the possum it could be done.\n\n- It was on its way to a poultry farmers\' convention.\n\nThe joke plays on the double meaning of "the other side" - literally crossing the road to the other side, or the "other side" meaning the afterlife. So it\'s an anti-joke, with a silly or unexpected pun as the answer.' additional_kwargs={} example=FalseWe can use our "LLM with Fallbacks" as we would a normal LLM.from langchain.prompts import ChatPromptTemplateprompt = ChatPromptTemplate.from_messages(    [        ("system", "You're a nice assistant who always includes a compliment in your response"),        ("human", "Why did the {animal} cross the road"),    ])chain = prompt | llmwith patch('openai.ChatCompletion.create', side_effect=RateLimitError()):    try:         print(chain.invoke({"animal": "kangaroo"}))    except:        print("Hit error")    content=" I don't actually know why the kangaroo crossed the road, but I can take a guess! Here are some possible reasons:\n\n- To get to the other side (the classic joke answer!)\n\n- It was trying to