cache_size_limit reached
I think that error it is something to do with KV.
Is there any method exist to empty KV cache manually?
Hi @uesenpai ,
Welcome to Google Gemma family of open source models, The KV cache can occupy a significant portion of memory and become a bottleneck for long-context generation. Memory efficient caches focus on trading off speed for reduced memory usage. This is especially important for large language models (LLMs) and if your hardware is memory constrained.
The OffloadedCache saves GPU memory by moving the KV cache for most model layers to the CPU. Only the current layer cache is maintained on the GPU during a models forward iteration over the layers. OffloadedCache asynchronously prefetches the next layer cache and sends the previous layer cache back to the CPU.
This cache strategy always generates the same result as DynamicCache and works as a drop-in replacement or fallback. You may want to use OffloadedCache if you have a GPU and you’re getting out-of-memory (OOM) errors.
The above mentioned mechanism can be utilized by passing the parameter cache_implementation="offloaded_static" while calling the generate() method or else you can pass use_cache=False to disable the cache.
Please find the attached screenshot for your reference.
If you required any further assistance please feel free to reach out to me.
Thanks.