It can speed up your application by reducing the number of API calls you make to the LLM provider.from langchain.globals import set_llm_cachefrom langchain.llms import OpenAI# To make the caching really obvious, lets use a slower model.llm = OpenAI(model_name="text-davinci-002", n=2, best_of=2)In Memory Cache‚Äãfrom langchain.cache import InMemoryCacheset_llm_cache(InMemoryCache())# The first time, it is not yet in cache, so it should take longerllm.predict("Tell me a joke")    CPU times: user 35.9 ms, sys: 28.6 ms, total: 64.6 ms    Wall time: 4.83 s    "\n\nWhy couldn't the bicycle stand up by itself? It was...two tired!"# The second time it is, so it goes fasterllm.predict("Tell me a joke")    CPU times: user 238 ¬µs, sys: 143 ¬µs, total: 381 ¬µs    Wall time: 1.76 ms    '\n\nWhy did the chicken cross the road?\n\nTo get to the other side.'SQLite Cache‚Äãrm .langchain.db# We can do the same thing with a SQLite cachefrom langchain.cache import SQLiteCacheset_llm_cache(SQLiteCache(database_path=".langchain.db"))# The first time, it is not yet in cache, so it should take longerllm.predict("Tell me a joke")    CPU times: user 17 ms, sys: 9.76 ms, total: 26.7 ms    Wall time: 825 ms    '\n\nWhy did the chicken cross the road?\n\nTo get to the other side.'# The second time it is, so it goes fasterllm.predict("Tell me a joke")    CPU times: user 2.46 ms, sys: 1.23 ms, total: 3.7 ms    Wall time: 2.67 ms    '\n\nWhy did the chicken cross the road?\n\nTo get to the other side.'Optional caching in chains‚ÄãYou can also turn off caching for particular nodes in chains. Note that because of certain interfaces, it's often easier to construct the chain first, and then edit the LLM afterwards.As an example, we will load a summarizer map-reduce chain. We will cache results for the map-step, but then not freeze it for the combine step.llm = OpenAI(model_name="text-davinci-002")no_cache_llm = OpenAI(model_name="text-davinci-002", cache=False)from langchain.text_splitter import
LangChain provides an optional caching layer for LLMs. This is useful for two reasons:
LangChain provides an optional caching layer for LLMs. This is useful for two reasons: ->: It can speed up your application by reducing the number of API calls you make to the LLM provider.from langchain.globals import set_llm_cachefrom langchain.llms import OpenAI# To make the caching really obvious, lets use a slower model.llm = OpenAI(model_name="text-davinci-002", n=2, best_of=2)In Memory Cache‚Äãfrom langchain.cache import InMemoryCacheset_llm_cache(InMemoryCache())# The first time, it is not yet in cache, so it should take longerllm.predict("Tell me a joke")    CPU times: user 35.9 ms, sys: 28.6 ms, total: 64.6 ms    Wall time: 4.83 s    "\n\nWhy couldn't the bicycle stand up by itself? It was...two tired!"# The second time it is, so it goes fasterllm.predict("Tell me a joke")    CPU times: user 238 ¬µs, sys: 143 ¬µs, total: 381 ¬µs    Wall time: 1.76 ms    '\n\nWhy did the chicken cross the road?\n\nTo get to the other side.'SQLite Cache‚Äãrm .langchain.db# We can do the same thing with a SQLite cachefrom langchain.cache import SQLiteCacheset_llm_cache(SQLiteCache(database_path=".langchain.db"))# The first time, it is not yet in cache, so it should take longerllm.predict("Tell me a joke")    CPU times: user 17 ms, sys: 9.76 ms, total: 26.7 ms    Wall time: 825 ms    '\n\nWhy did the chicken cross the road?\n\nTo get to the other side.'# The second time it is, so it goes fasterllm.predict("Tell me a joke")    CPU times: user 2.46 ms, sys: 1.23 ms, total: 3.7 ms    Wall time: 2.67 ms    '\n\nWhy did the chicken cross the road?\n\nTo get to the other side.'Optional caching in chains‚ÄãYou can also turn off caching for particular nodes in chains. Note that because of certain interfaces, it's often easier to construct the chain first, and then edit the LLM afterwards.As an example, we will load a summarizer map-reduce chain. We will cache results for the map-step, but then not freeze it for the combine step.llm = OpenAI(model_name="text-davinci-002")no_cache_llm = OpenAI(model_name="text-davinci-002", cache=False)from langchain.text_splitter import