LLM-dCache: Improving Tool-Augmented LLMs with GPT-Driven Localized Data Caching
<|reference_start|>LLM-dCache: Improving Tool-Augmented LLMs with GPT-Driven Localized Data Caching: As Large Language Models (LLMs) broaden their capabilities to manage thousands of API calls, they are confronted with complex data operations across vast datasets with significant overhead to the underlying system. In this work, we introduce LLM-dCache to optimize data accesses by treating cache operations as callable API functions exposed to the tool-augmented agent. We grant LLMs the autonomy to manage cache decisions via prompting, seamlessly integrating with existing function-calling mechanisms. Tested on an industry-scale massively parallel platform that spans hundreds of GPT endpoints and terabytes of imagery, our method improves Copilot times by an average of 1.24x across various LLMs and prompting techniques.<|reference_end|>
@article{singh2024llm-dcache:,
    title={LLM-dCache: Improving Tool-Augmented LLMs with GPT-Driven Localized Data
  Caching},
    author={Simranjit Singh, Michael Fore, Andreas Karatzas, Chaehong Lee, Yanan
  Jian, Longfei Shangguan, Fuxun Yu, Iraklis Anagnostopoulos, Dimitrios
  Stamoulis},
    journal={arXiv preprint arXiv:2406.06799},
    year={2024},
    archivePrefix={arXiv},
    eprint={2406.06799},
    primaryClass={cs.DC cs.CL}
}