## Focus, Distinguish, and Prompt: Unleashing CLIP for Efficient and Flexible Scene Text Retrieval

## Gangyan Zeng

## Yuan Zhang

weijin4@lenovo.com

School of Cyber Science and Engineering, Nanjing University of Science and Technology gyzeng@njust.edu.cn

State Key Laboratory of Media Convergence and Communication, Communication University of China yzhang@cuc.edu.cn

Jin Wei Lenovo Research

Dongbao Yang ∗ Institute of Information Engineering, Chinese Academy of Sciences yangdongbao@iie.ac.cn

## Peng Zhang ∗

School of Cyber Science and Engineering, Nanjing University of Science and Technology Laboratory for Advanced Computing and Intelligence Engineering zhang\_peng@njust.edu.cn

## Xugong Qin

School of Cyber Science and Engineering, Nanjing University of Science and Technology qinxugong@njust.edu.cn

Yiwen Gao School of Cyber Science and Engineering, Nanjing University of Science and Technology gaoyiwen@njust.edu.cn

## Yu Zhou

TMCC, College of Computer Science, Nankai University yzhou@nankai.edu.cn

## ABSTRACT

Scene text retrieval aims to find all images containing the query text from an image gallery. Current efforts tend to adopt an Optical Character Recognition (OCR) pipeline, which requires complicated text detection and/or recognition processes, resulting in inefficient and inflexible retrieval. Different from them, in this work we propose to explore the intrinsic potential of Contrastive Language-Image Pre-training (CLIP) for OCR-free scene text retrieval. Through empirical analysis, we observe that the main challenges of CLIP as a text retriever are: 1) limited text perceptual scale, and 2) entangled visual-semantic concepts. To this end, a novel model termed FDP (Focus, Distinguish, and Prompt) is developed. FDP first focuses on scene text via shifting the attention to the text area and probing the hidden text knowledge, and then divides the query text into content word and function word for processing, in which a semantic-aware prompting scheme and a distracted queries assistance module are utilized. Extensive experiments show that FDP significantly enhances the inference speed while achieving better or competitive retrieval accuracy compared to existing methods. Notably, on the IIIT-STR benchmark, FDP surpasses the state-of-the-art model by 4.37% with a 4 times faster speed. Furthermore, additional experiments under phrase-level and attribute-aware scene text retrieval

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from permissions@acm.org.

MM'24, October 28-November 1, 2024, Melbourne, VIC, Australia

ACM ISBN 979-8-4007-0686-8/24/10

© 2024 Copyright held by the owner/author(s). Publication rights licensed to ACM.

https://doi.org/10.1145/3664647.3680877

settings validate FDP's particular advantages in handling diverse forms of query text. The source code will be publicly available at https://github.com/Gyann-z/FDP.

## CCS CONCEPTS

- · Information systems → Multimedia and multimodal retrieval .

## KEYWORDS

Scene Text Retrieval; CLIP; Visual-Semantic Entanglement; Prompt Tuning

## ACMReference Format:

Gangyan Zeng, Yuan Zhang, Jin Wei, Dongbao Yang, Peng Zhang, Yiwen Gao, Xugong Qin and Yu Zhou. 2024. Focus, Distinguish, and Prompt: Unleashing CLIP for Efficient and Flexible Scene Text Retrieval. In Proceedings of the 32nd ACM International Conference on Multimedia (MM '24), October 28-November 1, 2024, Melbourne, VIC, Australia. ACM, New York, NY, USA, 10 pages. https://doi.org/10.1145/3664647.3680877

## 1 INTRODUCTION

Since text is ubiquitous in natural scenes and conveys rich semantic information, scene text understanding has received a lot of attention for decades [36, 46, 53]. Different from common scene text understanding tasks such as text detection [32, 33, 38, 44], text recognition [6, 30, 31, 56], and end-to-end text spotting [14, 19, 23, 43], Scene Text Retrieval (STR) is an emerging topic that only focuses on text of interest, i.e. , searching images containing a given query text from an image gallery. As such, STR is beneficial for many applications like product image search, program recommendation, and electronic book archives management [9, 10, 48].

*Corresponding authors.

Figure 1: Illustration of the trade-off between retrieval accuracy (mAP scores) and inference speed (FPS) on the IIIT-STR benchmark. Our proposed FDP method achieves better balance than previous methods.

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00441v1-figure-8ea335a8-58a5-401e-a3bf-2c5eb25fc408.jpg)

With the aid of Optical Character Recognition (OCR) techniques, STR has made remarkable progress in recent years [12, 15, 41]. Nevertheless, existing methods still suffer from two critical limitations. First, as illustrated in Fig.1, there is a dilemma of how to balance retrieval accuracy (mAP scores) and inference speed (FPS). Specifically, most STR models follow the two-stage pipeline that first detects text regions and then compares these regions with the query text for retrieval. In this pipeline, either an exact text detection or recognition process is required, which significantly slows down the inference speed. Comparatively, Gomez et al. [11] achieve fast text retrieval using a single-shot CNN architecture, but it is limited by relatively low retrieval accuracy. Second, in real life, the query text that people expect to retrieve is often in various forms. However, current efforts rely on the local retrieval mechanism that treats word instances as query units, leading to inherent inflexibility in phrase-level or attribute-aware scene text retrieval (see Fig.2).

Recently, Contrastive Language-Image Pre-training (CLIP) [34] has become a powerful foundation model for learning cross-modal representations and enabling zero-shot transfer to downstream tasks [7, 18, 21]. More remarkably, several works [20, 37] have demonstrated CLIP also implies OCR capabilities via pre-training on massive image-text pairs. It gives us a new insight: can we explore the intrinsic potential of CLIP for efficient and flexible STR? To this end, we investigate the advantages and deficiencies of CLIP in the STR task through an empirical study. A surprising finding is that simply applying the frozen CLIP can even achieve better accuracy than some dedicated STR models. Moreover, thanks to CLIP's simple network design, the retrieval speed is also superior. Despite these impressive results, there are still two challenges that hinder CLIP from being an ideal retrieval engine: 1) Limited text perceptual scale. As the image resolution input into CLIP is very limited ( e.g. , 224 × 224), and scene text usually occupies only a small part of the scene image, a lot of text may be ignored or misrecognized by CLIP. 2) Entangled visual-semantic concepts. Due to the prevalence of text in natural images, there is confusion between visual text and semantic concepts in CLIP's cognition [27]. Its specific impact on STR is that the CLIP-based retrieval model performs much better

Figure 2: Illustration of the scene text retrieval in (a) phraselevel and (b) attribute-aware settings. Unlike conventional STR models that rely on the local retrieval mechanism, FDP is more flexible in handling diverse forms of query text.

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00441v1-figure-30431ced-6f18-4cce-bbf9-f1e921889b04.jpg)

on content words ( e.g. , ' coffee ', ' hotel ') than on function words ( e.g. , ' and ', ' with ') because only content words represent exact semantics. Besides, the model may have difficulty distinguishing similar words ( e.g. , ' advice ' and ' advise ') because their semantics are close in the embedding space.

Overall, the main contributions of this work are three-fold: 1) To the best of our knowledge, it is the first work to directly extend CLIP for scene text retrieval. We summarize both the advantages and deficiencies of CLIP in dealing with the STR task and propose a novel FDP (Focus, Distinguish, and Prompt) method.

In this paper, we propose a model named FDP (Focus, Distinguish, and Prompt) to tackle the above challenges. Concretely, for each image in the gallery, we firstly force CLIP to focus on scene text by 1) applying the rough text localization results to refine the model attention on images, and 2) leveraging CLIP's well-aligned visionlanguage representations to prob text knowledge. Then, given a query text, we distinguish whether it is a content word or a function word via unsupervised clustering and determine the retrieval solution accordingly. Finally, a semantic-aware prompt ing scheme is developed, which converts the query text into a learnable prompt and ranks images by computing their similarity scores with each image. In addition, a distracted query assistance strategy is involved during training to resist the negative effects of similar words. Extensive experiments on three benchmarks show that FDP can achieve better or competitive accuracy compared to existing models with a faster inference speed. To further evaluate the effectiveness of STR methods over arbitrary-length query text, we introduce a new benchmark of phrase-level scene text retrieval (PSTR). Meanwhile, qualitative experiments regarding attribute-aware scene text retrieval are conducted. These experimental results demonstrate the generalization and flexibility of FDP.

2) In contrast to previous works, FDP steers the prior knowledge from CLIP and eliminates the complicated text detection/recognition process, thus achieving a better trade-off between retrieval accuracy and inference speed. Notably, FDP outperforms the state-of-the-art

method [47] by 4.37% mAP score with a 4 times faster speed on the IIIT-STR benchmark.

3) We evaluate existing STR methods in phrase-level and attributeaware scene text retrieval settings, further verifying the superiority of FDP in handling diverse forms of query text.

## 2 RELATED WORK

## 2.1 Scene Text Retrieval

Most of the early STR approaches tend to follow the OCR pipeline [1, 40, 50]. They first take two separate steps of text detection and recognition to extract words in each image, and then match these words with the query word for retrieval. For instance, Mishra et al. [28] first investigate the STR task, proposing to rank all images based on the ordering and positioning of localized characters. Jaderberg et al. [15] perform text spotting with a CNN network and evaluate the occurrences of the query word within the spotted words. However, those straightforward attempts could not obtain satisfactory performance and are also not efficient. To solve this problem, Gomez et al. [11] leverage a compact representation named Pyramidal Histogram of Character (PHOC) [2] and propose a single-shot CNN architecture that simultaneously predicts text proposals and corresponding PHOCs. In this way, the STR task can be completed by a simple nearest neighbor search. Considering current handcraft representations (including PHOC) still cannot well reflect the distance between text and image modalities, recent methods are dedicated to mining better similarity measures. TDSL [41] establishes an end-to-end network that jointly optimizes text detection and cross-modal similarity learning. To mitigate the gap across different modalities, Wen et al. [47] propose to cast STR as an image-to-image matching problem. Although better retrieval accuracy is achieved, it comes at the cost of inference speed.

## 2.2 Exploring CLIP's OCR Capabilities

Vision-language models pre-trained on web-scale data have been demonstrated to exhibit certain OCR capabilities [13, 24, 25, 49, 54]. As reported in [34], the CLIP model shows favorable OCR performance in rendered text images. To further mine the underlying rationales, [20] thoroughly inspects different versions of CLIP. This work uncovers that CLIP suffers from severe text spotting bias because many captions in CLIP's training dataset tend to parrot the visual text embedded within images. Through orthogonal projections of the learned representation space into 'learn to spell' and 'forget to spell' parts, [27] disentangles such bias to some extent. Besides, LoGoPrompt [37] finds that synthetic text images are good visual prompts for vision-language models, which can be used to improve image classification performance.

Inspired by these observations, several works aim to enhance OCR tasks by transferring knowledge from CLIP. In the field of scene text recognition, CLIP4STR [55] designs a two-branch framework in which the recognition results are predicted by the visual branch and then refined by the cross-modal branch. CLIP-OCR [45] resorts to the knowledge distillation technique and guides the recognition with both visual and linguistic knowledge from CLIP. In the field of scene text detection, TCM [51] integrates CLIP with existing text detectors, leading to obvious performance improvements in domain adaptation and few-shot capabilities. However,

CLIP merely acts as an auxiliary module in these works. Whether it is possible to turn CLIP directly into a scene text reader (spotter or retriever) remains an unexplored problem.

## 3 FDP METHOD

The overview of the proposed FDP framework is illustrated in Fig.3. Given a query text ( /u1D444 = ' house '), FDP fulfills the STR task with a pipeline of 'Focus, Distinguish, and Prompt'.

## 3.1 Focus

Considering CLIP is pre-trained on conventional image-text pairs and thus lacks fine-grained awareness of visual text information, the first step of FDP is directing CLIP to focus on scene text. To be specific, for each image from the gallery, we first square it to obtain the input image /u1D43C /u1D456 /u1D45B/u1D45D/u1D462 /u1D461 ∈ R /u1D43F × /u1D43F ( /u1D43F is the image length), i.e. , perform zero-padding to make the shorter side match the longer side. The goal is to avoid the loss of image content caused by the center cropping operation during CLIP's preprocessing. Then, the frozen ResNet-based vision encoder of CLIP is employed to extract the global feature /u1D453 /u1D454 /u1D459 /u1D45C /u1D44F /u1D44E/u1D459 ∈ R /u1D436 × /u1D43B × /u1D44A of /u1D43C /u1D456 /u1D45B/u1D45D/u1D462 /u1D461 , where /u1D436 , /u1D43B and /u1D44A stand for the channel, height and width dimensions respectively. Based on this global feature, two modules including dynamic attention shift and text knowledge probing are proposed to highlight scene text information and address the limited text perceptual scale problem. Dynamic Attention Shift. The limitation of input resolution is an intractable problem encountered by pre-trained vision-language models. It greatly impairs scene text understanding performance because text often occupies a very small part of the image. Existing efforts resolve this problem by subdividing into image patches [17], retraining a vision encoder [3], or processing in the frequency domain [8], which are not efficient. Instead, in this work we find that it may be enough to give the model a glimpse of the rough area where text is clustered. To this end, we employ text detection supervision to train a lightweight text localization network, and then utilize the normalized probability map to reweigh the features in the average pooling layer. Specifically, as the multi-head attention layer in CLIP's vision encoder loses the 2D image information, we first introduce a reformulated head following [57] to recover the 2D convolutional image feature /u1D453 /u1D450 /u1D45C /u1D45B /u1D463 ∈ R /u1D438 × /u1D43B × /u1D44A , where /u1D438 is the embedding dimension in CLIP. Then, the localization probability map /u1D43C /u1D459 /u1D45C /u1D450 ∈ R /u1D43B × /u1D44A is obtained via a learnable convolutional layer. We train the text localization network via a class-balanced crossentropy loss, given by:

$$\mathcal { L } _ { l o c } = - \beta Y l o g ( I _ { l o c } ) - ( 1 - \beta ) ( 1 - Y ) l o g ( 1 - I _ { l o c } ) \quad ( 1 ) \\ \dots \cdots \nu \colon \alpha \dots \dots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots$$

$$\pi _ { 0 } \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \j \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \ i \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \rangle \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \u \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \nolimits \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \eq \, \, \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \i \, \langle \, \i + \, \, \,$$

where /u1D44C is the ground-truth localization map generated by the text detection annotation, and /u1D6FD is a balancing factor defined as:

After that, the predicted localization probability map is adopted as a new attention mask to dynamically refine the attention applied to the global feature. The details of the dynamic attention shift module are illustrated in Fig.4. CLIP uses Transformer-style multi-head attention to perform average pooling, where the 2D global feature is first flattened into a 1D sequence and then generates a key-value pair to interact with the globally average-pooled feature (query).

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00441v1-figure-98641a6f-5e1d-4c6c-8518-fd71f85a6b24.jpg)

Figure 3: Overview of the proposed FDP model. It consists of three main parts: 1) Focus: Two main modules of dynamic attention shift and text knowledge probing are presented to highlight scene text information. 2) Distinguish: The query text is categorized into content words and function words via unsupervised clustering. 3) Prompt: The retrieval process is finally achieved by a semantic-aware prompting scheme, and meanwhile distracted queries are generated during training to assist in identifying similar words.

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00441v1-figure-8ad0b417-e6af-4272-af42-735e6b59a1fc.jpg)

encoder tend to activate on the whole image rather than specific text information. Therefore, we consider whether we could design a simple strategy to probe the text-related knowledge hidden in CLIP. Drawing inspiration from previous work [34] that conducts zero-shot image classification using a predefined template ' a photo of [CLS] ', we propose to utilize the plain text ' scene text ' as a probe and obtain its language embedding as the probe feature /u1D453 /u1D45D /u1D45F /u1D45C/u1D44F /u1D452 ∈ R /u1D438 , which is then interacted with /u1D453 /u1D44E/u1D461 /u1D461 /u1D45B . Since the representations of vision and language are well-aligned in the embedding space of CLIP, this probe will naturally turn CLIP into a model that is more sensitive to scene text. Subsequently, the interacted feature is fused with the attention feature /u1D453 /u1D44E/u1D461 /u1D461 /u1D45B as the image feature /u1D453 /u1D456/u1D45A/u1D454 to comprehensively encode the image for retrieval. The text knowledge probing process is formulated as:

Figure 4: Details of the dynamic attention shift module.

Consequently, the localization probability map is also flattened into a 1D sequence and then weights the global feature at each spatial location. The derived attention feature /u1D453 /u1D44E/u1D461 /u1D461 /u1D45B ∈ R /u1D438 can shift the model attention to the scene text area.

Text Knowledge Probing. Empirically, we find that when we query CLIP with the word ' house ', it is possible to return the corresponding object (an image of a house) instead of the scene text (an image says ' house '). This is because the neurons in CLIP's vision

$$\mu _ { \mu } \quad \mu _ { \mu } \quad f _ { i m g } = \text{MHCA} ( Q = f _ { a t t n }, K = f _ { p r o b e }, \mathbf V = f _ { p r o b e } ) + f _ { a t t n } \quad ( 3 )$$

where MHCA means the multi-head cross-attention mechanism.

## 3.2 Distinguish

Several works [4, 20] have revealed that the CLIP model exhibits inherent bias towards visual text, e.g. , an image of 'dog' may be recognized as 'cat' by placing the text that says 'cat'. The reason is that the captions CLIP pre-trained with are often simple repetitions of text embedded in images. In this work, we further observe that this bias is essentially the entanglement between visual and

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00441v1-figure-c827254b-8161-4d69-a3c5-e47b26ebb60f.jpg)

content words function words

Figure 5: Illustration of the effect caused by visual-semantic entanglement. (a) The t-SNE visualization of high-frequency scene text's CLIP language embeddings. (b) The comparison of the retrieval accuracy of three frozen CLIP models on content words and function words.

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00441v1-figure-e170ddf3-134f-4840-b9b4-f64159e70d83.jpg)

semantic concepts. To be specific, we select 500 words with the highest frequency from the MLT [29] training set, and then group their CLIP language embeddings into two clusters via K-Means. The t-SNE visualization result is depicted in Fig.5 (a). As can be seen, these words naturally fall into two clusters, namely, the content words and the function words. Between them, the content words have explicit semantics, usually appearing next to the thing they represent, so they exhibit strong visual-semantic entanglement. In contrast, the function words generally appear in the captions as conjunctions, which do not correspond to specific concepts. To investigate this effect on STR, we evaluate the retrieval accuracy of content words and function words respectively by applying the frozen CLIP models on SVT [42] dataset, as shown in Fig.5 (b). It is obvious that for the three CLIP models with different capacities, the mAP scores on function words are significantly lower than those on content words. This inspires us that different retrieval solutions should be taken on these two clusters. Thus, before each retrieval process, we pre-distinguish whether the given query is a content word or a function word. Without the need for specialized tools, this can be easily achieved via unsupervised clustering, namely predicting which K-Means cluster the query text belongs to.

## 3.3 Prompt

Prompt tuning is a promising paradigm that aims to adapt the knowledge from a pre-trained model to a target domain [22, 35, 52]. Mirroring the success of prompt tuning in natural language processing and cross-modal learning, we leverage it to facilitate CLIP for efficient text retrieval.

Semantic-aware Prompting. To improve retrieval performance on both content words and function words, we develop a semanticaware prompting scheme. Inspired by CoOp [58], we introduce two

Table 1: The setting of FDP models.

| Model   | Base model   |   Original input size |   Expanded input size |   /u1D438 |
|---------|--------------|-----------------------|-----------------------|-----------|
| FDP-S   | CLIP-RN50    |                   224 |                   640 |      1024 |
| FDP-B   | CLIP-RN50x4  |                   288 |                   720 |       640 |
| FDP-L   | CLIP-RN50x16 |                   384 |                   960 |       768 |

sets of learnable context vectors to serve the retrieval of content words and function words respectively. Formally, the text prompts fed to the frozen CLIP language encoder are organized as:

$$P _ { c } = [ p _ { c } ^ { 1 }, p _ { c } ^ { 2 }, \dots, p _ { c } ^ { M }, Q ] \quad \quad \quad ( 4 )$$

where /u1D443 /u1D450 and /u1D443 /u1D453 denote the prompts for content words and function words respectively. /u1D440 and /u1D441 are the length of learnable context vectors, and /u1D444 represents the query text.

$$P _ { f } = [ p _ { f } ^ { 1 }, p _ { f } ^ { 2 }, \dots, p _ { f } ^ { N }, Q ] \quad \quad ( 5 )$$

Then, the CLIP language encoder outputs the prompt feature /u1D453 /u1D45D /u1D45F /u1D45C/u1D45A/u1D45D /u1D461 . We calculate the cosine similarity between /u1D453 /u1D456/u1D45A/u1D454 and /u1D453 /u1D45D /u1D45F /u1D45C/u1D45A/u1D45D /u1D461 to measure the pairwise similarity score between the input image and query text, i.e. , /u1D446 ( /u1D43C , /u1D444 ) . The symmetric cross-entropy loss L /u1D44E/u1D459 /u1D456 /u1D454 /u1D45B over a batch is applied to contrastively align the matched ( /u1D43C , /u1D444 ) pairs.

Distracted Queries Assistance. Due to the limited input resolution and the visual-semantic entanglement, CLIP can perceive text to some extent, but it indeed lacks the ability of fine-grained character discrimination. To address this problem, a distracted queries assistance module is proposed, which teaches the FDP model to better recognize text during training. In particular, for a query text /u1D444 , we utilize a dictionary to generate /u1D43E (set to 5) distracted queries /u1D444 1 , /u1D444 2 , .../u1D444 /u1D43E that have the smallest edit distances with /u1D444 . These distracted queries are taken as hard negative samples which are also converted to text prompts and fed into the CLIP language encoder, predicting similarities scores /u1D446 ( /u1D43C , /u1D444 /u1D458 ) , /u1D458 = 1 , ..., /u1D43E . Meanwhile, the edit distance of each distracted query from the ground-truth query /u1D437 ( /u1D444 , /u1D444 /u1D458 ) is calculated. Subsequently, we convert these /u1D43E sets of similarity scores and edit distances into probability distributions and compute the KL divergence between them as the training loss L /u1D451 /u1D456 /u1D460 /u1D461 /u1D45F /u1D44E/u1D450 /u1D461 . The objective is to maximize the similarity scores of the distracted queries that are close to the ground-truth while minimizing the similarity scores of the distracted queries that are far from the ground-truth.

## 3.4 Optimization

The proposed FDP is trained with the following loss functions:

$$\mathcal { L } = \lambda _ { 1 } \mathcal { L } _ { l o c } + \lambda _ { 2 } \mathcal { L } _ { a l i g n } + \lambda _ { 3 } \mathcal { L } _ { d i s t r a c t } \quad \quad ( 6 ) \\ \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot$$

During inference, the distracted queries assistance module is removed. Given a query text /u1D444 , images in the gallery are ranked according to the predicted similarity score /u1D446 ( /u1D43C , /u1D444 ) . When extending our FDP model to phrase-level or attribute-aware scene text retrieval settings, /u1D444 is directly assigned the corresponding form of query, and the inference process remains unchanged.

where /u1D706 1, /u1D706 2, and /u1D706 3 are used to balance these losses, which are all set to 1 in our implementation.

Table 2: Comparison with existing methods on IIIT-STR, SVT, and TotalText benchmarks. * means the result with subdivision enhancement. Bold indicates the best performance, and underline indicates the second-best performance.

| Method                   | mAP      | mAP   | mAP       | FPS   |
|--------------------------|----------|-------|-----------|-------|
| Method                   | IIIT-STR | SVT   | TotalText | FPS   |
| Mishra et al. [28]       | 42.70    | 56.24 | -         | 0.10  |
| He et al. [12]           | 46.34    | 57.61 | -         | 2.35  |
| Jaderberg et al. [15]    | 66.50    | 86.30 | -         | 0.30  |
| ABCNet [23]              | 67.25    | 82.43 | 69.30     | 17.50 |
| Gomez et al. [11]        | 69.83    | 83.74 | -         | 43.50 |
| Mafla et al. [26]        | 71.67    | 85.74 | -         | 42.20 |
| Mask TextSpotter v3 [19] | 74.48    | 84.54 | 72.42     | 2.40  |
| TDSL [41]                | 77.09    | 89.38 | 74.75     | 12.00 |
| Wen et al. [47]          | 77.40    | 90.95 | 80.09     | 11.00 |
| CLIP-RN50                | 52.93    | 65.07 | 38.46     | 76.32 |
| CLIP-RN50x4              | 52.60    | 70.54 | 41.65     | 57.91 |
| CLIP-RN50x16             | 53.03    | 76.55 | 43.51     | 29.02 |
| FDP-S (Ours)             | 81.77    | 82.56 | 65.26     | 45.11 |
| FDP-B (Ours)             | 86.65    | 86.64 | 73.63     | 31.43 |
| FDP-L (Ours)             | 89.46    | 89.63 | 79.18     | 11.82 |
| FDP-L* (Ours)            | 91.49    | 91.18 | 82.02     | 3.04  |

## 4 EXPERIMENTS

## 4.1 Datasets

IIIT Scene Text Retrieval (IIIT-STR) [28] is a popular benchmark that contains 10000 images and 50 predefined queries. The images are collected using Google image search, so this dataset has a large variability in text fonts, styles, and viewpoints.

supported by CLIP is very limited, we expand the image size in FDP. However, directly expanding the image size makes the position embedding inherited from CLIP incompatible. To tackle it, we propose a new learnable position embedding whose parameters are initialized with the nearest interpolation of original parameters.

Street View Text (SVT) dataset [42] is a collection of natural street scenes. It consists of 100 training images and 249 testing images. All annotated words (427 words) in the test set are employed as the query text.

Multi-lingual Scene Text (MLT)-Eng is the English subset of MLT [29], which includes about 5000 images of natural scenes.

TotalText [5] is a scene text dataset consisting of 1255 training and 300 testing images. The 60 words that occur most frequently in the test set are selected as queries.

In our experiments, MLT-Eng is only used for training the proposed model. IIIT-STR, SVT, and TotalText are the testing datasets. It should be noted that as CLIP's potential is fully explored, 900k synthetic training images used in [41, 47] can be saved in FDP.

The query terms in existing datasets are all single words. To validate whether the STR models can be generalized to arbitrarylength query text, we introduce a new Phrase-level Scene Text Retrieval (PSTR) dataset. To build it, we select 36 phrases that occur frequently in life as queries, each containing 2 to 4 words, e.g. , ' school bus ', ' handle with care '. For each query, we collect 15 images from TextOCR dataset [39] and Google image search respectively. In total, PSTR includes 1080 images and 36 query text.

## 4.2 Implementation Details

Based on CLIP with different capacities, we build several versions of FDP models, as summarized in Tab.1. As the input image size

Weoptimize FDP using Adam [16] optimizer with an initial learning rate of 2e-3. The batch size is 48, and the number of training epochs is about 8. For fair comparisons, our experiments are implemented with Pytorch. All FDP models are trained on an NVIDIA A6000 GPU and tested on an NVIDIA 1080 GPU.

## 4.3 Comparison with Existing Methods

In this section, we compare FDP with existing methods on three STR benchmarks, i.e. , IIIT-STR, SVT, and TotalText. As a task to pursue practical applications, the inference speed of STR is undoubtedly very important, while previous methods are subject to the balance of retrieval accuracy and inference speed. In this paper, we first investigate employing the frozen CLIP model directly as the retrieval engine. As reported in Tab.2, it is surprising that CLIP already exhibits some retrieval capabilities even though it has not been specially trained on STR tasks. In particular, CLIP-RN50 obtains 52.93% and 65.07% mAP scores on the IIIT-STR and SVT datasets respectively, which even exceeds several dedicated STR models [12, 28] at a much faster speed (76.32 FPS).

Based on this observation, FDP is proposed to better unleash CLIP's potential for the STR task. On the IIIT-STR benchmark, we can notice that FDP-S initialized with the CLIP-RN50 base model boosts the mAP score by 28.84% (52.93%-&gt;81.77%), achieving an appealing result of 81.77%. Meanwhile, the inference speed is also superior (45.11 FPS), even faster than PHOC-based methods [11, 26]. When upgrading the model to a large size, FDP-L significantly outperforms the competitive model [47] by 12.06% (77.40%-&gt;89.46%)

Table 3: Ablation experiments on IIIT-STR and SVT datasets.

|    | Focus                   | Focus                  |             | Prompt                   | Prompt                        |          |       |
|----|-------------------------|------------------------|-------------|--------------------------|-------------------------------|----------|-------|
| #  | Dynamic Attention Shift | Text Knowledge Probing | Distinguish | Semantic-aware Prompting | Distracted Queries Assistance | IIIT-STR | SVT   |
| 1  | /enc-37                 | /enc-37                | /enc-37     | /enc-37                  | /enc-37                       | 75.74    | 79.97 |
| 2  | /check                  | /enc-37                | /enc-37     | /enc-37                  | /enc-37                       | 78.38    | 80.21 |
| 3  | /check                  | /check                 | /enc-37     | /enc-37                  | /enc-37                       | 78.93    | 80.27 |
| 4  | /check                  | /check                 | /enc-37     | vanilla                  | /enc-37                       | 80.07    | 81.03 |
| 5  | /check                  | /check                 | /check      | /check                   | /enc-37                       | 81.27    | 81.94 |
| 6  | /check                  | /check                 | /check      | /check                   | /check                        | 81.77    | 82.56 |

Table 4: Analysis of the context length on SVT benchmark.

|   /u1D440 |     2 |     4 |     8 |
|-----------|-------|-------|-------|
|         2 | 81.8  | 81.71 | 80.65 |
|         4 | 82.56 | 82.11 | 81.68 |
|         8 | 82.01 | 81.44 | 80.86 |

Table 5: Comparison with existing methods on PSTR dataset.

| Method            |   mAP |   FPS |
|-------------------|-------|-------|
| Gomez et al. [11] | 68.01 | 42.45 |
| TDSL [41]         | 89.4  | 11.34 |
| FDP-S (Ours)      | 92.28 | 45.11 |

at a comparable speed. Compared with IIIT-STR, the query terms of SVT and TotalText contain many function words and often occupy small areas in images, which are more challenging for STR models. Nevertheless, even without a complicated network design, FDP also reaches competitive performance on these datasets. To further boost the retrieval accuracy, we attempt to integrate the subdivision enhancement strategy here, i.e. , subdividing the input image into 4 equal patches and combing the outputs of these patches. The mAP scores are improved by 2.03%, 1.55%, and 2.84% on IIIT-STR, SVT, and TotalText, outperforming existing STR methods.

the dynamic attention shift and text knowledge probing modules can be considered to highlight scene text information from visual space and semantic space respectively. They bring 2.64% and 0.55% gains on the IIIT-STR dataset, which are proven to be effective. In particular, as IIIT-STR contains a large number of images without any text, the 'Focus' step has a more significant effect on the IIITSTR dataset than on the SVT dataset. Then, we study the effect of different prompt strategies. When simply adopting the learnable prompt method in [58] (#4), the mAP scores reach 80.07% and 81.03% on these two datasets. In contrast, we claim that content words and function words should be distinguished and exploit different customized prompts. Following this idea, our semantic-aware prompting scheme improves the performance to 81.27% and 81.94%. Further, by adding the training strategy of distracted queries assistance, 81.77% and 82.56% mAP scores are finally obtained.

To provide intuitive analyses of FDP in comparison with previous methods, a typical example is visualized in Fig.6 (a). Given the query word ' adobe ', TDSL relies entirely on character composition for retrieval. If the scene text is blurry or small, it can easily be misrecognized. Besides, text-like patterns tend to interfere with model decisions. Instead, our FDP model takes full advantage of visual context information, returning the desired images from an image gallery. From the rank@7 and rank@10 images retrieved by FDP-S, we notice that the proposed method can recall images where the query text is not so salient.

## 4.4 Ablation Study

Overall results. In Tab.3, a detailed ablation experiment is conducted to verify the effectiveness of each module. We start by training a model that only utilizes the new learnable position embedding, whose mAP scores on IIIT-STR and SVT are 75.74% and 79.97% respectively. It reveals that enlarging the image size ( i.e. , enhancing the text perceptual scale) is of critical importance for STR. Based on this, we gradually add the proposed modules and observe that each module brings noticeable improvements. In the 'Focus' step,

Analysis of the context length. In the semantic-aware prompting module, the hyperparameters /u1D440 and /u1D441 determine the context length for content words and function words respectively. As shown in Tab.4, we evaluate the model performance on the SVT dataset to analyze the effect of these hyperparameters. According to the results, FDP reaches the best performance when /u1D440 = 4 and /u1D441 = 2. It may suggest that function words contain less semantics than content words, so they do not require complicated descriptions of context. More ablation experiment results can be found in the supplementary material.

## 4.5 Extending to More Retrieval Settings

Phrase-level scene text retrieval. In reality, the scene text that people expect to retrieve is often of arbitrary length, such as ' ice cream ', ' do it yourself '. To validate the generality of our method over arbitrary-length query text, we evaluate FDP and several STR models on PSTR. For the comparison models [11, 41], since they can only accept word instances, we split each phrase into words, search them separately, and then average the corresponding similarity scores. As shown in Tab.5, FDP is more flexible than existing STR models in phrase-level retrieval. Specifically, the PHOC-based method [11] only achieves 68.01% mAP score on PSTR. We speculate this is because many split words are too short ( e.g. , ' do ' in ' do it yourself ') to be accurately retrieved by the PHOC-based search algorithm. Although TDSL [41] can get 89.40% with the simple splitting operation, it is inherently flawed due to the local retrieval mechanism. From Fig.6 (b), we can see that for the query text ' no smoking ', TDSL may return images containing ' no engine ' (rank@3) or ' no softener ' (rank@5), which do not meet retrieval expectations. In addition, due to the dense text distribution in the PSTR dataset, these

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00441v1-figure-a994568e-cbab-4945-af3b-998399793a1e.jpg)

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00441v1-figure-a85bc626-3932-4362-a567-47d559491f9b.jpg)

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00441v1-figure-bf52d547-0767-4122-afac-3ec536d23293.jpg)

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00441v1-figure-37c5c07c-9e84-46b9-b8cd-db323b0ab372.jpg)

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00441v1-figure-8334bd2b-0d19-4a23-a1a5-9425284bf2bb.jpg)

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00441v1-figure-a8c505ed-c0ec-40d8-8c0f-4fc1722508bb.jpg)

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00441v1-figure-64439014-5870-4427-868b-a09092e6ce41.jpg)

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00441v1-figure-f653ebd9-b48f-45d4-ac64-79c60d14103d.jpg)

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00441v1-figure-e1267630-b946-4cb5-917f-f1b910ba6977.jpg)

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00441v1-figure-a8184bba-042a-44e1-9117-505354ab3b0f.jpg)

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00441v1-figure-82658e23-be50-4237-85f4-47776366e5ed.jpg)

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00441v1-figure-112645dd-e5c0-4237-9648-acd975b16804.jpg)

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00441v1-figure-9c0a7c0e-043c-40de-9579-177d4c818252.jpg)

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00441v1-figure-81ea0bd5-2c00-4b74-9916-956f92aebcaf.jpg)

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00441v1-figure-3e760e7c-a48c-45c3-9f06-bd10ae80084c.jpg)

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00441v1-figure-5b00c4a3-d350-41a6-a81e-da40c2d9e88c.jpg)

Figure 6: Visualization of retrieval results. (a) An example on IIIT-STR benchmark, in which rank@6-10 retrieval results are provided. (b) An example on PSTR benchmark, in which rank@1-5 retrieval results are provided. The correct results are highlighted in green while the incorrect ones are highlighted in red. Best viewed in zoom.

| #   | Benchmark   | Query          | Method   | Retrieval Results   |
|-----|-------------|----------------|----------|---------------------|
| (a) | IIIT-STR    | ' adobe '      | TDSL     |                     |
| (a) | IIIT-STR    | ' adobe '      | FDP-S    |                     |
| (b) | PSTR        | ' no smoking ' | TDSL     |                     |
| (b) | PSTR        |                | FDP-S    |                     |

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00441v1-figure-6e7e4e5d-2d55-404c-a7bd-6d4e3f533fe2.jpg)

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00441v1-figure-2633dab7-b25e-432c-9516-a5242d2eb1a3.jpg)

OCR-based comparison models run slower than on IIIT-STR. In contrast, the FDP-S model reaches 92.28% mAP score on PSTR, outperforming existing methods by great margins. More importantly, as FDP does not rely on text detection or recognition processes, the retrieval speed will not be affected.

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00441v1-figure-d95beb8d-296f-43ed-a01f-1d74ca25a5d3.jpg)

Attribute-aware scene text retrieval. Considering that people often query scene text with fine-grained attributes for more accurate search results, we further explore extending FDP to the attributeaware scene text retrieval setting. We design some attribute-related queries and search corresponding images from the IIIT-STR dataset. Several typical retrieval examples are illustrated in Fig.7. These results manifest that the CLIP-based FDP model is naturally suitable for attribute-aware scene text retrieval because it takes advantage of CLIP's prior knowledge. FDP can well deal with attribute-related information such as color, font, and even position of scene text, returning images that users want. Admittedly, this is not available for conventional OCR-based STR models.

## 5 CONCLUSION

| Query                          | Retrieval Results   |
|--------------------------------|---------------------|
| ' dairy in white '             |                     |
| ' galaxy in red background '   |                     |
| ' coffee in italic font '      |                     |
| ' school written on the wall ' |                     |

In this paper, we explore CLIP's intrinsic potential for efficient and flexible scene text retrieval. An OCR-free retrieval model named FDP (Focus, Distinguish, and Prompt) is proposed, in which the 'Focus' design highlights scene text information hidden in CLIP while 'Distinguish' and 'Prompt' designs further overcome the negative effects caused by visual-semantic entanglement. Experimental results on three datasets demonstrate the effectiveness of our proposed modules and show that FDP achieves a better tradeoff between retrieval accuracy and inference speed. In addition, FDP can easily generalize to other settings like phrase-level and

Figure 7: Qualitative examples of attribute-aware scene text retrieval. Best viewed in zoom.

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00441v1-figure-d7c8e89c-31dc-4730-ad1a-f4986281b78f.jpg)

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00441v1-figure-9d348cc2-0416-43a9-9198-521672d5681d.jpg)

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00441v1-figure-d4bc2205-bdf1-437c-ae5c-21ec0f196f5a.jpg)

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00441v1-figure-4b76e3be-208c-410c-bf3f-5936b8867793.jpg)

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00441v1-figure-cca6c651-f8b3-4e34-8bc3-543ff41c6247.jpg)

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00441v1-figure-43ce6218-3a01-45bc-92ce-a406d18959be.jpg)

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00441v1-figure-05c63cea-5dee-4207-9cd9-62faa04f700e.jpg)

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00441v1-figure-0d7028c1-2b95-4941-b8ba-2c21103cf375.jpg)

attribute-aware scene text retrieval, which are more practical for requirements in real scenarios.

## ACKNOWLEDGMENTS

This work is supported by the National Natural Science Foundation of China (Grant NO 62376266), the National Key R &amp; D Program of China (No.2022YFB3103800) and the fund of Laboratory for Advanced Computing and Intelligence Engineering.

Focus, Distinguish, and Prompt: Unleashing CLIP for Efficient and Flexible Scene Text Retrieval

## REFERENCES

- [26] Andrés Mafla, Ruben Tito, Sounak Dey, Lluís Gómez, Marçal Rusiñol, Ernest Valveny, and Dimosthenis Karatzas. 2021. Real-time lexicon-free scene text retrieval. PR 110 (2021), 107656.
- [2] Jon Almazán, Albert Gordo, Alicia Fornés, and Ernest Valveny. 2014. Word spotting and recognition with embedded attributes. TPAMI 36, 12 (2014), 25522566.
- [4] Liangliang Cao, Bowen Zhang, Chen Chen, Yinfei Yang, Xianzhi Du, Wencong Zhang, Zhiyun Lu, and Yantao Zheng. 2023. Less is more: Removing text-regions improves clip training efficiency and robustness. arXiv preprint arXiv:2305.05095 (2023).
- [3] Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. 2023. Qwen-vl: A frontier large visionlanguage model with versatile abilities. arXiv preprint arXiv:2308.12966 (2023).
- [5] Chee Kheng Ch'ng and Chee Seng Chan. 2017. Total-text: A comprehensive dataset for scene text detection and recognition. In ICDAR , Vol. 1. 935-942.
- [7] Bo Fang, Wenhao Wu, Chang Liu, Yu Zhou, Yuxin Song, Weiping Wang, Xiangbo Shu, Xiangyang Ji, and Jingdong Wang. 2023. UATVR: Uncertainty-adaptive text-video retrieval. In ICCV . 13723-13733.
- [6] Yongkun Du, Zhineng Chen, Caiyan Jia, Xiaoting Yin, Tianlun Zheng, Chenxia Li, Yuning Du, and Yu-Gang Jiang. 2022. SVTR: Scene Text Recognition with a Single Visual Model. In IJCAI . 884-890.
- [8] Hao Feng, Qi Liu, Hao Liu, Wengang Zhou, Houqiang Li, and Can Huang. 2023. DocPedia: Unleashing the Power of Large Multimodal Model in the Frequency Domain for Versatile Document Understanding. arXiv preprint arXiv:2311.11810 (2023).
- [10] Suman K Ghosh and Ernest Valveny. 2015. Query by string word spotting based on character bi-gram indexing. In ICDAR . 881-885.
- [9] Suman K Ghosh, Lluis Gomez, Dimosthenis Karatzas, and Ernest Valveny. 2015. Efficient indexing for query by string text retrieval. In ICDAR . 1236-1240.
- [11] Lluís Gómez, Andrés Mafla, Marçal Rusinol, and Dimosthenis Karatzas. 2018. Single shot scene text retrieval. In ECCV . 700-715.
- [13] Anwen Hu, Haiyang Xu, Jiabo Ye, Ming Yan, Liang Zhang, Bo Zhang, Chen Li, Ji Zhang, Qin Jin, Fei Huang, et al. 2024. mPLUG-DocOwl 1.5: Unified Structure Learning for OCR-free Document Understanding. arXiv preprint arXiv:2403.12895 (2024).
- [12] Tong He, Zhi Tian, Weilin Huang, Chunhua Shen, Yu Qiao, and Changming Sun. 2018. An end-to-end textspotter with explicit alignment and attention. In CVPR . 5020-5029.
- [14] Mingxin Huang, Yuliang Liu, Zhenghao Peng, Chongyu Liu, Dahua Lin, Shenggao Zhu, Nicholas Yuan, Kai Ding, and Lianwen Jin. 2022. Swintextspotter: Scene text spotting via better synergy between text detection and text recognition. In CVPR . 4593-4603.
- [16] Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. In ICLR . 4190-4198.
- [15] Max Jaderberg, Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. 2016. Reading text in the wild with convolutional neural networks. IJCV 116, 1 (2016), 1-20.
- [17] Zhang Li, Biao Yang, Qiang Liu, Zhiyin Ma, Shuo Zhang, Jingxu Yang, Yabo Sun, Yuliang Liu, and Xiang Bai. 2024. Monkey: Image resolution and text label are important things for large multi-modal models. In CVPR . 26763-26773.
- [19] Minghui Liao, Guan Pang, Jing Huang, Tal Hassner, and Xiang Bai. 2020. Mask textspotter v3: Segmentation proposal network for robust scene text spotting. In ECCV . 706-722.
- [18] Feng Liang, Bichen Wu, Xiaoliang Dai, Kunpeng Li, Yinan Zhao, Hang Zhang, Peizhao Zhang, Peter Vajda, and Diana Marculescu. 2023. Open-vocabulary semantic segmentation with mask-adapted clip. In CVPR . 7061-7070.
- [20] Yiqi Lin, Conghui He, Alex Jinpeng Wang, Bin Wang, Weijia Li, and Mike Zheng Shou. 2023. Parrot Captions Teach CLIP to Spot Text. arXiv preprint arXiv:2312.14232 (2023).
- [22] Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. 2021. Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing. arXiv preprint arXiv:2107.13586 (2021).
- [21] Ziyi Lin, Shijie Geng, Renrui Zhang, Peng Gao, Gerard De Melo, Xiaogang Wang, Jifeng Dai, Yu Qiao, and Hongsheng Li. 2022. Frozen clip models are efficient video learners. In ECCV . 388-404.
- [23] Yuliang Liu, Hao Chen, Chunhua Shen, Tong He, Lianwen Jin, and Liangwei Wang. 2020. ABCNet: Real-time scene text spotting with adaptive bezier-curve network. In CVPR . 9809-9818.
- [25] Tengchao Lv, Yupan Huang, Jingye Chen, Lei Cui, Shuming Ma, Yaoyao Chang, Shaohan Huang, Wenhui Wang, Li Dong, Weiyao Luo, et al. 2023. Kosmos-2.5: A multimodal literate model. arXiv preprint arXiv:2309.11419 (2023).
- [24] Yuliang Liu, Zhang Li, Hongliang Li, Wenwen Yu, Mingxin Huang, Dezhi Peng, Mingyu Liu, Mingrui Chen, Chunyuan Li, Lianwen Jin, et al. 2023. On the hidden mystery of ocr in large multimodal models. arXiv preprint arXiv:2305.07895 (2023).
- [1] David Aldavert, Marcal Rusinol, Ricardo Toledo, and Josep Lladós. 2013. Integrating visual and textual cues for query-by-string word spotting. In ICDAR . 511-515.
- [28] Anand Mishra, Karteek Alahari, and CV Jawahar. 2013. Image retrieval using textual cues. In ICCV . 3040-3047.
- [27] Joanna Materzyńska, Antonio Torralba, and David Bau. 2022. Disentangling visual and written concepts in clip. In CVPR . 16410-16419.
- [29] Nibal Nayef, Yash Patel, Michal Busta, Pinaki Nath Chowdhury, Dimosthenis Karatzas, Wafa Khlif, Jiri Matas, Umapada Pal, Jean-Christophe Burie, Cheng-lin Liu, et al. 2019. ICDAR2019 robust reading challenge on multi-lingual scene text detection and recognition-RRC-MLT-2019. In ICDAR . 1582-1587.
- [31] Zhi Qiao, Yu Zhou, Dongbao Yang, Yucan Zhou, and Weiping Wang. 2020. SEED: Semantics enhanced encoder-decoder framework for scene text recognition. In CVPR . 13528-13537.
- [30] Zhi Qiao, Yu Zhou, Jin Wei, Wei Wang, Yuan Zhang, Ning Jiang, Hongbin Wang, and Weiping Wang. 2021. PIMNet: a parallel, iterative and mimicking network for scene text recognition. In ACM MM . 2046-2055.
- [32] Xugong Qin, Pengyuan Lyu, Chengquan Zhang, Yu Zhou, Kun Yao, Peng Zhang, Hailun Lin, and Weiping Wang. 2023. Towards robust real-time scene text detection: From semantic to instance representation learning. In ACM MM . 20252034.
- [34] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021. Learning transferable visual models from natural language supervision. In ICML . 8748-8763.
- [33] Xugong Qin, Yu Zhou, Youhui Guo, Dayan Wu, Zhihong Tian, Ning Jiang, Hongbin Wang, and Weiping Wang. 2021. Mask is all you need: Rethinking mask r-cnn for dense and arbitrary-shaped scene text detection. In ACM MM . 414-423.
- [35] Yongming Rao, Wenliang Zhao, Guangyi Chen, Yansong Tang, Zheng Zhu, Guan Huang, Jie Zhou, and Jiwen Lu. 2022. Denseclip: Language-guided dense prediction with context-aware prompting. In CVPR . 18082-18091.
- [37] Cheng Shi and Sibei Yang. 2023. Logoprompt: Synthetic text images can be good visual prompts for vision-language models. In ICCV . 2932-2941.
- [36] Xuejian Rong, Chucai Yi, and Yingli Tian. 2022. Unambiguous Text Localization, Retrieval, and Recognition for Cluttered Scenes. TPAMI 44, 3 (2022), 1638-1652.
- [38] Yan Shu, Wei Wang, Yu Zhou, Shaohui Liu, Aoting Zhang, Dongbao Yang, and Weipinng Wang. 2023. Perceiving ambiguity and semantics without recognition: an efficient and effective ambiguous scene text detector. In ACMMM . 1851-1862.
- [40] Sebastian Sudholt and Gernot A Fink. 2016. Phocnet: A deep convolutional neural network for word spotting in handwritten documents. In ICFHR . 277-282.
- [39] Amanpreet Singh, Guan Pang, Mandy Toh, Jing Huang, Wojciech Galuba, and Tal Hassner. 2021. TextOCR: Towards large-scale end-to-end reasoning for arbitraryshaped scene text. In CVPR . 8802-8812.
- [41] Hao Wang, Xiang Bai, Mingkun Yang, Shenggao Zhu, Jing Wang, and Wenyu Liu. 2021. Scene text retrieval via joint text detection and similarity learning. In CVPR . 4558-4567.
- [43] Pengfei Wang, Chengquan Zhang, Fei Qi, Shanshan Liu, Xiaoqiang Zhang, Pengyuan Lyu, Junyu Han, Jingtuo Liu, Errui Ding, and Guangming Shi. 2021. Pgnet: Real-time arbitrarily-shaped text spotting with point gathering network. In AAAI , Vol. 35. 2782-2790.
- [42] Kai Wang, Boris Babenko, and Serge Belongie. 2011. End-to-end scene text recognition. In ICCV . 1457-1464.
- [44] Wei Wang, Yu Zhou, Jiahao Lv, Dayan Wu, Guoqing Zhao, Ning Jiang, and Weipinng Wang. 2022. Tpsnet: Reverse thinking of thin plate splines for arbitrary shape scene text representation. In ACM MM . 5014-5025.
- [46] Jin Wei, Yuan Zhang, Yu Zhou, Gangyan Zeng, Zhi Qiao, Youhui Guo, Haiying Wu, Hongbin Wang, and Weipinng Wang. 2022. Textblock: Towards scene text spotting without fine-grained detection. In ACM MM . 5892-5902.
- [45] Zixiao Wang, Hongtao Xie, Yuxin Wang, Jianjun Xu, Boqiang Zhang, and Yongdong Zhang. 2023. Symmetrical Linguistic Feature Distillation with CLIP for Scene Text Recognition. In ACM MM . 509-518.
- [47] Lilong Wen, Yingrong Wang, Dongxiang Zhang, and Gang Chen. 2023. Visual Matching is Enough for Scene Text Retrieval. In WSDM . 447-455.
- [49] Jiabo Ye, Anwen Hu, Haiyang Xu, Qinghao Ye, Ming Yan, Guohai Xu, Chenliang Li, Junfeng Tian, Qi Qian, Ji Zhang, et al. 2023. Ureader: Universal ocr-free visually-situated language understanding with multimodal large language model. arXiv preprint arXiv:2310.05126 (2023).
- [48] Xiao Yang, Dafang He, Wenyi Huang, Alexander Ororbia, Zihan Zhou, Daniel Kifer, and C Lee Giles. 2017. Smart library: Identifying books on library shelves using supervised deep learning for scene text reading. In JCDL . 1-4.
- [50] Xu-Cheng Yin, Xuwang Yin, Kaizhu Huang, and Hong-Wei Hao. 2013. Accurate and robust text detection: A step-in for text retrieval in natural scene images. In SIGIR . 1091-1092.
- [52] Gangyan Zeng, Yuan Zhang, Yu Zhou, Bo Fang, Guoqing Zhao, Xin Wei, and Weiping Wang. 2023. Filling in the blank: Rationale-augmented prompt tuning for TextVQA. In ACM MM . 1261-1272.
- [51] Wenwen Yu, Yuliang Liu, Wei Hua, Deqiang Jiang, Bo Ren, and Xiang Bai. 2023. Turning a CLIP Model into a Scene Text Detector. In CVPR . 6978-6988.

- [53] Gangyan Zeng, Yuan Zhang, Yu Zhou, Xiaomeng Yang, Ning Jiang, Guoqing Zhao, Weiping Wang, and Xu-Cheng Yin. 2023. Beyond OCR+ VQA: Towards end-to-end reading and reasoning for robust and accurate textvqa. PR 138 (2023), 109337.
- [55] Shuai Zhao, Xiaohan Wang, Linchao Zhu, and Yi Yang. 2023. CLIP4STR: A Simple Baseline for Scene Text Recognition with Pre-trained Vision-Language Model.
- [54] Jiarui Zhang, Jinyi Hu, Mahyar Khayatkhoei, Filip Ilievski, and Maosong Sun. 2024. Exploring Perceptual Limitation of Multimodal Large Language Models. arXiv preprint arXiv:2402.07384 (2024).

arXiv preprint arXiv:2305.14014 (2023).

- [57] Chong Zhou, Chen Change Loy, and Bo Dai. 2022. Extract free dense labels from clip. In ECCV . 696-712.
- [56] Tianlun Zheng, Zhineng Chen, Shancheng Fang, Hongtao Xie, and Yu-Gang Jiang. 2024. Cdistnet: Perceiving multi-domain character distance for robust text recognition. IJCV 132, 2 (2024), 300-318.
- [58] Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. 2022. Learning to prompt for vision-language models. IJCV 130, 9 (2022), 2337-2348.

## Focus, Distinguish, and Prompt: Unleashing CLIP for Efficient and Flexible Scene Text Retrieval (Supplementary Material)

## 1 PSTR DATASET

To validate whether the STR models can be generalized to arbitrarylength query text, we introduce a new Phrase-level Scene Text Retrieval (PSTR) dataset. Specifically, we select 36 phrases that occur frequently in life as queries, each containing 2 to 4 words. All queries are listed in Tab.1.

Analysis of the number of K-Means clusters. To verify the reasonability of distinguishing the query text into content word and function word, we vary the number of K-Means clusters /u1D70E on SVT dataset and report the corresponding results in Tab.3. From the results, FDP-S performs best when /u1D70E = 2, indicating that using more clusters do not necessarily lead to better results.

Table 1: The list of queries in PSTR dataset.

|   Length | Queries                                                                                                                                                                                                                                                                                                                                                                                    |
|----------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
|        2 | ' apple store ', ' bud light ', ' coming soon ', ' low price ', ' no smoking ', ' one way ', ' school bus ', ' second edition ', ' upper canada ', ' brewing company ', ' fly emirates ', ' blue moon ', ' nutrition facts ', ' cabernet sauvignon ', ' fitting room ', ' ice cream ', ' joe boxer ', ' macbook air ', ' caps lock ', ' san francisco ', ' sony ericsson ', ' the original |
|        3 | ' bank of america ', ' do it yourself ', ' handle with care ', ' happy new year ', ' made in china ', ' india pale ale ', ' olive oil soap ', ' pedro benito urbina ', ' slide to unlock '                                                                                                                                                                                                 |
|        4 | ' in god we trust ', ' share a coke with ', ' have a nice day ', ' pink lady apple juice ', ' single malt scotch whisky '                                                                                                                                                                                                                                                                  |

## 2 MORE ABLATION STUDIES

Analysis of the predefined probe. The goal of the predefined probe is to stimulate the text-related knowledge hidden in CLIP. In Tab.2, we conduct an ablation study of the predefined probe on the IIIT-STR benchmark. The results show that if the predefined probe is removed, the mAP score decreases from 81.77% to 79.58%. Furthermore, different strings are utilized to generate language embeddings that interact with the image attention feature. Compared to the 'without predefined probe' baseline, these text-related probes can enhance the performance. Among them, 'scene text' contributes to the best accuracy, implying that in CLIP's training data, the plain text 'scene text' may appear frequently with the scene text content from images.

Table 2: Analysis of the predefined probe on IIIT-STR benchmark.

| Predefined probe          |   mAP |
|---------------------------|-------|
| without predefined probe  | 79.58 |
| 'text'                    | 80.79 |
| 'word'                    | 80.84 |
| 'a set of text instances' | 80.41 |
| 'scene text'              | 81.77 |

Table 3: Analysis of the number of K-Means clusters on SVT benchmark.

| /u1D70E   |     1 |     2 |     3 |     5 |
|-----------|-------|-------|-------|-------|
| mAP       | 81.76 | 82.56 | 81.37 | 79.58 |

Analysis of the number of distracted queries. In the distracted queries assistance module, /u1D43E distracted queries are generated to help the model identify similar words. The ablation results of /u1D43E are reported in Tab.4. From them, we can see that a smaller /u1D43E may weaken the discrimination ability of FDP, while a larger /u1D43E will introduce many negative samples that are far from the query. Thus, /u1D43E is set to 5 in our experiments.

Table 4: Analysis of the number of distracted queries on IIITSTR benchmark.

| /u1D43E   |     3 |     5 |     7 |    10 |
|-----------|-------|-------|-------|-------|
| mAP       | 81.51 | 81.77 | 81.75 | 81.61 |

Analysis of the loss weights. The hyperparameters /u1D706 1, /u1D706 2, and /u1D706 3 are used to balance the loss items L /u1D459 /u1D45C /u1D450 , L /u1D44E/u1D459 /u1D456 /u1D454 /u1D45B , and L /u1D451 /u1D456 /u1D460 /u1D461 /u1D45F /u1D44E/u1D450 /u1D461 respectively in training FDP. Considering L /u1D44E/u1D459 /u1D456 /u1D454 /u1D45B is the main loss for contrastively aligning the matched (Image, Query) pairs, we set /u1D706 2 = 1 and conduct ablation studies of /u1D706 1 and /u1D706 3 on IIIT-STR dataset. The results are reported in Tab.5 and Tab.6. According to the ablation results, FDP reaches the best performance when /u1D706 1 = 1 and /u1D706 3 = 1.

Table 5: Ablation of the hyperparameter /u1D706 1 when /u1D706 3 = 1 on IIIT-STR benchmark.

| /u1D706 1   |   0.5 |     1 |     3 |     5 |
|-------------|-------|-------|-------|-------|
| mAP         | 81.33 | 81.77 | 80.98 | 80.46 |

Table 6: Ablation of the hyperparameter /u1D706 3 when /u1D706 1 = 1 on IIIT-STR benchmark.

| /u1D706 3   |   0.5 |     1 |     3 |    5 |
|-------------|-------|-------|-------|------|
| mAP         |  81.6 | 81.77 | 81.73 | 80.2 |

## 3 COMPARISON OF OCR-BASED AND OCR-FREE METHODS

To better demonstrate the superiority of our OCR-free STR framework, we compare it with a CLIP-based OCR pipeline. Specifically, this pipeline leverages TCM for text detection and CLIP-OCR for text recognition, which performs retrieval considering edit distances between the given query and spotted words. As shown in Tab.7, the OCR-based pipeline reaches 72.45% mAP score on IIITSTR, lagging behind FDP-S by 9.32%. A case study is provided in Fig.1, from which we can observe that, compared to the OCR-based STR pipeline, FDP-S avoids the accumulation of errors from detection and recognition, thus ranking the images reasonably.

Table 7: Performance comparison between the OCR-based method and OCR-free method on IIIT-STR benchmark.

| Method                   |   mAP |
|--------------------------|-------|
| OCR-based (TCM+CLIP-OCR) | 72.45 |
| OCR-free (FDP-S)         | 81.77 |

Query:

' google '

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00441v1-figure-0edc5f04-9024-4a95-8cd7-6746efe2eb49.jpg)

(a)

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00441v1-figure-84bdecbb-7a0a-46ca-8b35-51c4484cb2b8.jpg)

(b)

OCR-based (TCM+CLIP-OCR) OCR-free (FDP-S)

rank@9     (sim=1.46e-04)

rank@304  (sim=1.23e-04)

rank@531 (sim=2.88e-05)

rank@7      (sim=8.23e-03)

Figure 1: A case study for OCR-based vs. OCR-free methods. The correct result is highlighted in green while the incorrect one is highlighted in red. Best viewed in zoom.

## 4 MORE QUALITATIVE RESULTS

To further support our claim in the paper, we provide more qualitative examples retrieved by FDP-S in Fig.2. The proposed FDP method could deal with scene text in various scenarios, and has ability to recalling complicated cases such as multi-oriented and curve text. In particular, with the aid of our semantic-aware prompting technique, the retrieval accuracy on the function words (such as ' the ') is significantly strengened compared to the original CLIP model. Nevertheless, for the challenging SVT and TotalText benchmarks, FDP still suffers from some limitations. On the one hand, whenthe query text to be retrieved is extremely tiny and meanwhile there are many disturbing words appearing in the image, the model has difficulty locating the target text. On the other hand, FDP still tends to return the images containing the similar words ( e.g. , ' port ' vs. ' sport ', ' since ' vs. ' venice ') from the image gallery, suggesting that the ability of fine-grained character discrimination needs to be further improved. We would like to go on with exploration for addressing these problems in the future.

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00441v1-figure-31021fc5-8c3c-4a84-a005-0bc05e78538d.jpg)

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00441v1-figure-91ad1ffd-8462-4aa3-9434-a29cd852ca08.jpg)

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00441v1-figure-f3e89bdb-54a5-4126-b4fd-89a2ce36331e.jpg)

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00441v1-figure-e8fc3888-11c7-430b-b0c7-3ba3f85649b1.jpg)

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00441v1-figure-fce8e7f0-a1ac-49f0-b775-a6863cbd7e2b.jpg)

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00441v1-figure-cb8a6d21-41aa-4afa-b8a6-d2f4402b1614.jpg)

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00441v1-figure-5cba0d04-00e0-4241-b777-453da203268c.jpg)

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00441v1-figure-56be83b2-4327-4f15-9d2d-9f85ab60e9a9.jpg)

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00441v1-figure-4a9352ac-732e-4fb3-acd2-341a5d86dcd8.jpg)

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00441v1-figure-204ad1bf-619b-4eb7-9ba4-31451f829a76.jpg)

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00441v1-figure-eb47302d-39cd-42fb-99cb-98e74a40098f.jpg)

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00441v1-figure-c3cff33c-4171-4f4b-9e23-fe213ba7d340.jpg)

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00441v1-figure-241ddd83-2974-4541-b0d0-245169c3aa7c.jpg)

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00441v1-figure-4ec450c1-bfa4-4fca-ae78-5f54250d3590.jpg)

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00441v1-figure-15badbec-2bd0-48d9-b5e3-e56bb3085175.jpg)

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00441v1-figure-ef571177-a4fd-4b95-a29b-3fcbcef7b1ca.jpg)

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00441v1-figure-05779b61-a53b-4fb2-8de1-f6783f567fa6.jpg)

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00441v1-figure-5431452e-5f42-4c5c-99aa-ad1367c35c1b.jpg)

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00441v1-figure-12b6819f-29b6-43da-8298-647df79cc954.jpg)

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00441v1-figure-139dbf70-f235-4823-8ddb-c2b653b238b0.jpg)

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00441v1-figure-94c0861d-1cad-49ae-817d-b66fe20c99ac.jpg)

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00441v1-figure-45db62e2-dd95-4e6a-bbbf-c32e5eb77e33.jpg)

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00441v1-figure-4c65e1c8-2faa-4305-83a5-09ddc2bd20e5.jpg)

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00441v1-figure-794bb711-f800-41af-9e7e-686ddc5ee0d5.jpg)

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00441v1-figure-2f2d74aa-865c-48f6-8f52-7524426814e8.jpg)

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00441v1-figure-1bc362ad-b207-4f07-8cf6-5f3f2131035c.jpg)

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00441v1-figure-099714fc-5546-4e11-ba2c-d98e7d61d01b.jpg)

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00441v1-figure-aaf8ff3d-8f30-40a9-beb1-e0fcffa8d3b2.jpg)

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00441v1-figure-e6811225-3aab-4d4b-bdd1-aeac0fa2cb22.jpg)

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00441v1-figure-0ccd472a-af82-4bc5-9ca8-269af93ad712.jpg)

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00441v1-figure-7a68dcb3-4aca-40d0-8220-a2ae22bec647.jpg)

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00441v1-figure-11e35eea-4fad-4625-a79e-88b8a57a549b.jpg)

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00441v1-figure-cda8c3ac-a410-4726-a181-6bd3dc51a645.jpg)

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00441v1-figure-f74fb162-94fc-4c33-bd41-4c15db7dc848.jpg)

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00441v1-figure-c35788ec-8ff2-4438-8690-06fec727954d.jpg)

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00441v1-figure-33b2e221-9165-4a7a-8d1d-992645a60442.jpg)

| Benchmark   | Query               | Retrieval Results   |
|-------------|---------------------|---------------------|
| IIIT-STR    | ' free '            |                     |
| IIIT-STR    | ' department '      |                     |
| SVT         | ' street '          |                     |
| SVT         | ' the '             |                     |
| TotalText   | ' since '           |                     |
| TotalText   | ' port '            |                     |
| PSTR        | ' bank of america ' |                     |
| PSTR        | ' have a nice day ' |                     |

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00441v1-figure-d0444a15-83f8-48ea-abe7-de09b9a05467.jpg)

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00441v1-figure-e7f1e85d-ecad-439f-b7af-0b5d6ec8d6ec.jpg)

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00441v1-figure-29faa151-083a-44db-9639-0ae034367d48.jpg)

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00441v1-figure-7bcf8b34-1740-4df5-9597-46333c7496b9.jpg)

Figure 2: Visualization of the rank@1-5 retrieval results from FDP-S on IIIT-STR, SVT, TotalText and PSTR benchmarks. The correct results are highlighted in green while the incorrect ones are highlighted in red. Best viewed in zoom.
Gangyan Zeng Yuan Zhang Jin Wei Dongbao Yang Peng Zhang Yiwen Gao Xugong Qin Yu Zhou
Focus, Distinguish, and Prompt: Unleashing CLIP for Efficient and Flexible Scene Text Retrieval
Scene text retrieval aims to find all images containing the query text from
an image gallery. Current efforts tend to adopt an Optical Character
Recognition (OCR) pipeline, which requires complicated text detection and/or
recognition processes, resulting in inefficient and inflexible retrieval.
Different from them, in this work we propose to explore the intrinsic potential
of Contrastive Language-Image Pre-training (CLIP) for OCR-free scene text
retrieval. Through empirical analysis, we observe that the main challenges of
CLIP as a text retriever are: 1) limited text perceptual scale, and 2)
entangled visual-semantic concepts. To this end, a novel model termed FDP
(Focus, Distinguish, and Prompt) is developed. FDP first focuses on scene text
via shifting the attention to the text area and probing the hidden text
knowledge, and then divides the query text into content word and function word
for processing, in which a semantic-aware prompting scheme and a distracted
queries assistance module are utilized. Extensive experiments show that FDP
significantly enhances the inference speed while achieving better or
competitive retrieval accuracy compared to existing methods. Notably, on the
IIIT-STR benchmark, FDP surpasses the state-of-the-art model by 4.37% with a 4
times faster speed. Furthermore, additional experiments under phrase-level and
attribute-aware scene text retrieval settings validate FDP's particular
advantages in handling diverse forms of query text. The source code will be
publicly available at https://github.com/Gyann-z/FDP.