  Deep prompt tuning (DPT) has gained great success in most natural language
processing~(NLP) tasks. However, it is not well-investigated in dense retrieval
where fine-tuning~(FT) still dominates. When deploying multiple retrieval tasks
using the same backbone model~(e.g., RoBERTa), FT-based methods are unfriendly
in terms of deployment cost: each new retrieval model needs to repeatedly
deploy the backbone model without reuse. To reduce the deployment cost in such
a scenario, this work investigates applying DPT in dense retrieval. The
challenge is that directly applying DPT in dense retrieval largely
underperforms FT methods. To compensate for the performance drop, we propose
two model-agnostic and task-agnostic strategies for DPT-based retrievers,
namely retrieval-oriented intermediate pretraining and unified negative mining,
as a general approach that could be compatible with any pre-trained language
model and retrieval task. The experimental results show that the proposed
method (called DPTDR) outperforms previous state-of-the-art models on both
MS-MARCO and Natural Questions. We also conduct ablation studies to examine the
effectiveness of each strategy in DPTDR. We believe this work facilitates the
industry, as it saves enormous efforts and costs of deployment and increases
the utility of computing resources. Our code is available at
https://github.com/tangzhy/DPTDR.

Computation and Language;Information Retrieval