Bottleneck-Minimal Indexing for Generative Document Retrieval
<|reference_start|>Bottleneck-Minimal Indexing for Generative Document Retrieval: We apply an information-theoretic perspective to reconsider generative document retrieval (GDR), in which a document $x \in X$ is indexed by $t \in T$, and a neural autoregressive model is trained to map queries $Q$ to $T$. GDR can be considered to involve information transmission from documents $X$ to queries $Q$, with the requirement to transmit more bits via the indexes $T$. By applying Shannon's rate-distortion theory, the optimality of indexing can be analyzed in terms of the mutual information, and the design of the indexes $T$ can then be regarded as a {\em bottleneck} in GDR. After reformulating GDR from this perspective, we empirically quantify the bottleneck underlying GDR. Finally, using the NQ320K and MARCO datasets, we evaluate our proposed bottleneck-minimal indexing method in comparison with various previous indexing methods, and we show that it outperforms those methods.<|reference_end|>
@article{du2024bottleneck-minimal,
    title={Bottleneck-Minimal Indexing for Generative Document Retrieval},
    author={Xin Du, Lixin Xiu, Kumiko Tanaka-Ishii},
    journal={arXiv preprint arXiv:2405.10974},
    year={2024},
    archivePrefix={arXiv},
    eprint={2405.10974},
    primaryClass={cs.IR cs.AI cs.CL cs.LG}
}