PIR: Remote Sensing Image-Text Retrieval with Prior Instruction Representation Learning
<|reference_start|>PIR: Remote Sensing Image-Text Retrieval with Prior Instruction Representation Learning: Remote sensing image-text retrieval constitutes a foundational aspect of remote sensing interpretation tasks, facilitating the alignment of vision and language representations. This paper introduces a prior instruction representation (PIR) learning paradigm that draws on prior knowledge to instruct adaptive learning of vision and text representations. Based on PIR, a domain-adapted remote sensing image-text retrieval framework PIR-ITR is designed to address semantic noise issues in vision-language understanding tasks. However, with massive additional data for pre-training the vision-language foundation model, remote sensing image-text retrieval is further developed into an open-domain retrieval task. Continuing with the above, we propose PIR-CLIP, a domain-specific CLIP-based framework for remote sensing image-text retrieval, to address semantic noise in remote sensing vision-language representations and further improve open-domain retrieval performance. In vision representation, Vision Instruction Representation (VIR) based on Spatial-PAE utilizes the prior-guided knowledge of the remote sensing scene recognition by building a belief matrix to select key features for reducing the impact of semantic noise. In text representation, Language Cycle Attention (LCA) based on Temporal-PAE uses the previous time step to cyclically activate the current time step to enhance text representation capability. A cluster-wise Affiliation Loss (AL) is proposed to constrain the inter-classes and to reduce the semantic confusion zones in the common subspace. Comprehensive experiments demonstrate that PIR could enhance vision and text representations and outperform the state-of-the-art methods of closed-domain and open-domain retrieval on two benchmark datasets, RSICD and RSITMD.<|reference_end|>
@article{pan2024pir:,
    title={PIR: Remote Sensing Image-Text Retrieval with Prior Instruction
  Representation Learning},
    author={Jiancheng Pan, Muyuan Ma, Qing Ma, Cong Bai and Shengyong Chen},
    journal={arXiv preprint arXiv:2405.10160},
    year={2024},
    archivePrefix={arXiv},
    eprint={2405.10160},
    primaryClass={cs.CV cs.AI}
}