CREST: Effectively Compacting a Datastore For Retrieval-Based Speculative Decoding
<|reference_start|>CREST: Effectively Compacting a Datastore For Retrieval-Based Speculative Decoding: We present CREST (Compact Retrieval-Based Speculative Decoding), a redesign of REST that allows it to be effectively "compacted". REST is a drafting technique for speculative decoding based on retrieving exact n-gram matches of the most recent n tokens generated by the target LLM from a datastore. The key idea of CREST is to only store a subset of the smallest and most common n-grams in the datastore with the hope of achieving comparable performance with less storage space. We found that storing a subset of n-grams both reduces storage space and improves performance. CREST matches REST's accepted token length with 10.6-13.5x less storage space and achieves a 16.5-17.1% higher acceptance length than REST using the same storage space on the HumanEval and MT Bench benchmarks.<|reference_end|>
@article{ho2024crest:,
    title={CREST: Effectively Compacting a Datastore For Retrieval-Based
  Speculative Decoding},
    author={Sophia Ho, Jinsol Park, Patrick Wang},
    journal={arXiv preprint arXiv:2408.04678},
    year={2024},
    archivePrefix={arXiv},
    eprint={2408.04678},
    primaryClass={cs.CL cs.AI cs.DB}
}