## DNTextSpotter: Arbitrary-Shaped Scene Text Spotting via Improved Denoising Training

Yu Xie 1 2 , * Qian Qiao 1 2 , * Jun Gao 3 Tianxiang Wu 1 Jiaqing Fan 1 † Yue Zhang 1 Jielei Zhang 2 Huyang Sun 2 1 Soochow University 2 bilibili Inc. 3 The Hong Kong Polytechnic University xieyu20001003@gmail.com, qqiao@stu.suda.edu.cn, { zhangjielei, sunhuyang @bilibili.com } https://qianqiaoai.github.io/projects/dntextspotter/

## Abstract

More and more end-to-end text spotting methods based on Transformer architecture have demonstrated superior performance. These methods utilize a bipartite graph matching algorithm to perform one-to-one optimal matching between predicted objects and actual objects. However, the instability of bipartite graph matching can lead to inconsistent optimization targets, thereby affecting the training performance of the model. Existing literature applies denoising training to solve the problem of bipartite graph matching instability in object detection tasks. Unfortunately, this denoising training method cannot be directly applied to text spotting tasks, as these tasks need to perform irregular shape detection tasks and more complex text recognition tasks than classification. To address this issue, we propose a novel denoising training method (DNTextSpotter) for arbitrary-shaped text spotting. Specifically, we decompose the queries of the denoising part into noised positional queries and noised content queries. We use the four Bezier control points of the Bezier center curve to generate the noised positional queries. For the noised content queries, considering that the output of the text in a fixed positional order is not conducive to aligning position with content, we employ a masked character sliding method to initialize noised content queries, thereby assisting in the alignment of text content and position. To improve the model's perception of the background, we further utilize an additional loss function for background characters classification in the denoising training part. Although DNTextSpotter is conceptually simple, it outperforms the state-of-the-art methods on four benchmarks (Total-Text, SCUT-CTW1500, ICDAR15, and Inverse-Text), especially yielding an improvement of

* Equal contribution. †Corresponding author. A part of this work was done during Yu Xie and Qian Qiao's internship at Bilibili Inc.

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00355v3-figure-5b9fe64a-823d-4cb9-9ad9-a3c06fdc4ba5.jpg)

iteration

Figure 1. The convergence curves of DNTextSpotter (Ours), DeepSolo, TESTR, and ESTextSpotter on the Inverse-Text dataset using the ResNet-50 backbone in the 'None' results, where 'None' denotes the F1-measure without lexicon.

11 3% . against the best approach in Inverse-Text dataset.

## 1. Introduction

Text Spotter, as an essential foundational technology that encompasses text detection and recognition, plays a critical role in various domains [6, 32, 39, 49] such as autonomous driving, security monitoring, and social media analysis. Given that textual elements in wild scenarios are often set against complex backgrounds, presented in diverse font sizes, and subject to distortions, accurate identification of text remains a challenging and dynamic field of research. To address these challenges, traditional CNN-based text spotters [1, 23, 26, 29, 36, 40, 42] divide the text localization task into separate detection and recognition stages, following a detect-then-recognize principle.

Compared to these classical spotting algorithms, building on the Deformable DETR [54], TESTR [52] represents a significant advancement in text spotting with its dual decoder design. This novel approach streamlines the process by removing the necessity for manually designed components and eliminating intermediate steps. TESTR simplifies the complex tasks of detection and recognition by treating them as a unified set prediction problem. It leverages bipartite graph matching to concurrently assign labels for both detection and recognition, achieving a more efficient and integrated workflow. Despite the significant achievements of TESTR, the employment of two decoders significantly increases computational complexity. Moreover, initializing queries with distinct properties for detection and recognition poses challenges for model optimization. Many methods have been proposed to address these issues. For example, TTS [18] attempts to unify the detection and recognition tasks within a single decoder. DeepSolo [48], while using a single decoder, introduces a novel query form with shared parameters, which initializes decoder queries by utilizing a series of explicit coordinates generated from the text line. However, despite further improvements in performance, these methods overlook the instability introduced by the bipartite matching employed in the DETR-like methods. In general object detection tasks, DN-DETR [19] firstly points out the instability problem of bipartite graph matching when using the DETR architecture and proposes denoising training to solve this problem. Denoising training, in simple terms, initializes noised queries using ground truth with a small amount of noise added, allowing for direct loss calculation with the ground truth after decoding, bypassing the bipartite graph matching algorithm. DINO [50] further proposes a contrastive denoising training method to further enhance the performance of denoising training. Unfortunately, for the tasks of spotting scene text, the challenge is significantly amplified due to the arbitrary shapes of the text to be detected and the need for recognition tasks that are more complex than mere classification. This complexity makes it difficult to directly apply this denoising training method. In fact, ESTextSpotter [15] directly incorporates a DINO-based denoising training method within its model architecture. In this context, as shown by the convergence curves in Fig. 1, this denoising training starts with regular bounding boxes as queries initialization, and the results on inverse-like texts become very poor, indirectly reflecting the negative impact brought by this coarse prior.

In this paper, we propose a novel denoising training method specifically designed for transformer-based text spotters that handle arbitrary shapes. Considering that the task of text spotting aims at the detection and recognition of text in any shape, and using regular boxes to initialize noised queries is coarse, we abandon the traditional approach that relies on 4D anchor boxes and classification labels. Instead, we use Bezier control points and text characters to initialize noised queries. Technically, we feed the noised queries obtained from the ground truth along with randomly initialized learnable queries into the decoder. We design the noised query using bezier control points of the bezier center curve and text scripts, thereby accomplishing the denoising of both points coordinates and text characters. In addition, considering that outputting text characters in a fixed positional order is not conducive to aligning position with content, we use a masked character sliding method to initialize noised content queries before initializing the text script as noised queries. This method assists in aligning the content and position of the text instance.

We verify the effectiveness of DNTextSpotter by using multiple public datasets. In the metrics of 'None' results with ResNet-50 backbone, compared with the current stateof-the-art methods, our method achieves 2 0% . and 2 1% . improved results on the Total Text and CTW1500 datasets respectively, reaching 84 5% . and 67 0% . respectively. On the newly released benchmark Inverse-Text dataset, our method even exceeds the state-of-the-art results by 11 3% . , reaching 75 9% . . When switching to the ViTAEv2-S backbone, scores for all metrics are further improved.

Our main contributions can be summarized as follows:

- · We introduce a novel denoising training method to design an end-to-end text spotting architecture. Starting from the attribute of arbitrary shapes of scene text, we utilize bezier control points as well as text characters to design this denoising training method.
- · Taking into account the negative impact of directly using ground truth text scripts to initialize noised queries, which leads to misalignment between the position of the characters and the content of these characters, we design a masked character sliding method to preprocess these ground truth text scripts, thereby optimizing the alignment between text position and content.
- · Our method achieves state-of-the-art results on multiple benchmarks. Specifically, we conduct a qualitative analysis of several text spotting architectures based on the transformer structure, including analyses of instability results and visualization of results.

## 2. Related Works

Early literature tends to classify end-to-end text spotting architecture into two-stage methods and one-stage methods. Recently, due to the popularity of transformer-based text spotters, we categorize these methods into CNN-based methods and transformer-based methods. Earlier comprehensive surveys on text spotting are available in [6, 32].

## 2.1. CNN-based Text Spotter

The first end-to-end scene text recognition network [20] combines detection and recognition into a single system.

This method is limited to recognizing regular-shaped text. Subsequent works [1, 10] improve the connection between the detector and recognizer, considering single characters or text blocks to handle irregular text more flexibly. The Mask TextSpotter series [23, 33] employs segmentation approaches to generate proposals. These methods rely on character-level annotations, significantly increasing the effort required for generating ground truth. Text Perceptron [36] and Boundary [40] utilize Thin-Plate-Spline [4] transformation to rectify features obtained from curved text. The ABCNet series [27,29] use BezierAlign to address the problem of curved text, requiring the prediction of a small fixed number of points.

While these methods achieve good performance, they require additional RoI-based [11] or TPS-based connectors, and the only shared part between the detector and recognizer is the backbone network's features, neglecting the collaborative nature of detection and recognition. [53] propose ARTS, highlighting the importance of collaborative detection and recognition in the text spotting task. Lastly, all of the aforementioned methods require complex manual operations like Non-Maximum Suppression (NMS).

## 2.2. Transformer-based Text Spotter

With the impressive success of Transformers [38] in visual tasks [8, 9, 17, 21, 30, 31, 46, 51], also influenced by the DETR family [5, 19, 25, 34, 50, 54], more recent works explore Transformer-based structures for the Text Spotting. TESTR [52] employs dual decoders for detection and recognition tasks, sharing the backbone and Transformer encoder features. TTS [18] utilizes an encoder and a decoder with multiple prediction heads for performing multitasks. DeepSolo [48] employs an explicit points method to model decoder queries.

Although these methods achieve promising results, they still exhibit certain limitations. The random initialized queries used in TESTR [52] and TTS [18] still lack clarity and fail to efficiently represent queries encompassing both positional and semantic aspects. Utilizing the encoder's output features, DeepSolo [48] generates Bezier center curve proposals, which subsequently serve to generate positional queries. It effectively decouples the ambiguously defined queries into positional queries and content queries. Although these methods have achieved certain accomplishments, the use of bipartite graph matching algorithms to obtain one-to-one matching results has been proven to have a negative impact on detection tasks by DN-DETR [19] and DINO [50]. Based on this, ESTextSpotter [15] attempted to use the denoising training method [50]. However, this method uses boxes as a positional prior for point prediction, failing to consider the irregular attributes of text instances and the characteristics of text scripts. Therefore, we design a denoising training ap- proach based on Bezier control points and characters.

## 3. Methodology

## 3.1. Preliminaries

Denoising Training. The denoising training method was first proposed by DN-DETR to address the slow convergence issue of DETR. This method constructs an additional auxiliary task in the decoder section without bipartite matching, which can be used to accelerate the convergence of DETR-like methods. Technically, it additionally feeds noised ground-truth boxes and labels into the transformer decoder to reconstruct these ground-truths, and this part is updated through an additional auxiliary DN loss. DINO improved the denoising training method and further proposed a contrastive denoising training method, which adds negative queries in addition to the original noised queries to predict the background. In our method, we further extend the denoising training approach based on this foundation.

Bezier Center Curve. ABCNet [27] was the first to use Bezier curves to flexibly adapt to any shape of scene text with a small number of fixed points. Subsequently, DeepSolo introduced the Bezier Center Curve, which initializes transformer decoder queries by uniformly sampling a fixed number of points on the curve. This Bezier Center Curve is obtained by calculating the average of the four Bezier control points on the top and bottom edges of each text instance. In our method, we consider utilizing Bezier center curves in denoising training.

## 3.2. Query Initialization

The ambiguous meaning of decoder queries in DETR is often interpreted in existing literature [25, 50] as a combination of positional queries and content queries. We also utilize this modeling approach and follow DeepSolo, using the Bezier Center Curve to initialize positional queries and then combine them with learnable content queries. In the denoising part, we represent noised queries as two parts: noised positional queries and noised content queries.

Noised Positional Queries. As shown in Fig. 3, for any text instance, the four Bezier control points of the Bezier center curve. Assuming that a picture contains N text instances, we can represent the set of all instances as: S = { S j | j = 1 2 , , . . . , N } , S j is a set of the four Bezier control points of the Bezier center curve of a text instance. Each instance can be represented as:

$$\mathcal { S } _ { j } = \{ ( x _ { i j }, y _ { i j } ) | i = 0, 1, 2, 3 \}, \quad \quad ( 1 )$$

where each point is composed of a pair of coordinates ( x ij , y ij ) , and the index i ranges from 0 to 3 , indexing the four Bezier control points. Random noise is added to these

Figure 2. The overall framework of DNTextSpotter. The model utilizes a backbone and an encoder to extract multi-scale features. The queries of the decoder can be divided into two parts: a matching part and a denoising part. The queries in the matching part are randomly initialized queries. The noised queries of denoising part can be found in Fig .3 and Fig .4. After decoder and task-specific head, the matching part calculates loss through a bipartite graph matching algorithm, and the denoising part calculates loss directly with the ground truth.

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00355v3-figure-1fb20033-614c-4717-8355-bc4cb744f9a5.jpg)

Figure 3. We generate noised positional queries using four Bezier control points from the ground truth, which includes uniformly sampling points along the Bezier curve, position embedding, and a two-layer MLP.

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00355v3-figure-41a57337-38cf-471c-b60c-f4cc86474a42.jpg)

points to obtain:

$$\mathcal { S } _ { j } ^ { ^ { \prime } } = \{ ( x _ { i j } + \Delta x _ { i j }, y _ { i j } + \Delta y _ { i j } ) | i = 0, 1, 2, 3 \}, \quad ( 2 ) \quad \text{slidi}$$

where ∆ x ij and ∆ y ij are obtained by calculating the distance between the four Bezier control points of the center curve and the four Bezier control points on the top side and are denoted as D x ij and D y ij . We use elements α ij and β ij from sets that satisfy a 0 -1 uniform distribution to control the noise ratio for the x and y coordinates respectively. Therefore, we can represent the offset of the coordinates as follows:

$$\Delta x _ { i j } = \begin{cases} \begin{array} { c c } ( - 1 ) ^ { m } \alpha _ { i j } D _ { i j } ^ { x } & \text{if positive;} \\ ( - 1 ) ^ { m } ( \alpha _ { i j } + 1 ) D _ { i j } ^ { x } & \text{otherwise.} \end{array} \end{cases} ( 3 ) \quad \text{one}$$

$$\Delta y _ { i j } = \begin{cases} \ ( - 1 ) ^ { m } \beta _ { i j } D _ { i j } ^ { y } & \text{if positive;} \\ ( - 1 ) ^ { m } ( \beta _ { i j } + 1 ) D _ { i j } ^ { y } & \text{otherwise.} \end{cases} \quad ( 4 ) \quad \text{acte}$$

where m and n are used to control the direction of the offset, with 'positive' indicating the coordinates of the positive part.

After adding noise to the Bezier control points, we obtain a new set of control points S ′ , Using these noisy Bezier control points, we uniformly sample T point coordinates from the resulting Bezier curves. These point coordinates form tensor Points with shape ( N,T, 2) . Finally, these coordinates are processed through positional encoding ( PE ) and two layers of MLP to obtain Noised Positional Queries ( Q N ) with a shape of ( N,T, 256) . We represent Q N as follows:

$$Q _ { N } = M L P ( P E ( P o i n t s ) ). \text{ \quad \ \ } ( 5 )$$

Section 4.4 analyzes 'Why add noise to the Bezier control points?'.

Noised Content Queries. We designed Mask Character Sliding (MCS), as shown in Fig. 4, to initialize the noised content queries. For the maximum recognition length T , which is equal to the T mentioned above, we perform a sliding operation on the valid characters in the positive part. Specifically, we first determine the number of valid characters t , which refers to the number of characters in the input sequence that actually have meaning. Then, we calculate the number of times each valid character should be cloned by performing a division operation ⌊ T t ⌋ , in order to evenly distribute the total length T of the sequence to each valid character. Additionally, since the division of T by t may not be exact, there will be a remainder k = T - ⌊ T t ⌋ , indicating that there are k additional spaces that need to be allocated. To fairly distribute these extra spaces, we assign one additional clone to each of the first k valid characters in the sequence, ensuring that the allocation for each character is as even as possible. Since this operation visually resembles a character sliding operation, we name it Char-

Figure 4. This image shows the process of generating noised content queries from the ground-truth texts. ' ∅ ' indicates the characters will be masked, while ' σ ' denotes flipping the characters into any character.

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00355v3-figure-964a7f25-6290-4f67-a23d-2f0e6ec2e8b0.jpg)

acter Sliding. After sliding the characters, we use a mask operation to control the number of consecutive characters, flipping a portion of the consecutive characters into a background label with a certain probability. After processing the positive part, we add noise to the characters in both the positive part and the negative part, causing these characters to flip to other characters with a probability of λ . These characters are then transformed into noised content queries after being embedded, with all the characters initialized in the negative part being backgrounds.

In addition, we use a dynamic group g to fully utilize the performance of denoising training. Considering the computational cost, we set the maximum number of text instances N per image to 100 . When N exceeds 100, we simply use the slicing method to take the first 100 instances. The division of g is as follows:

$$g = ( 5, \left \lfloor \frac { 1 0 0 } { n } \right \rfloor ). \quad \quad \quad ( 6 )$$

## 3.3. Single Attention Mask

To ensure that during the decoder self-attention calculation, the information in the denoising part contains groundtruth information, we need to ensure that the information in the matching part cannot see the information in the denoising part. In addition, each group should not be able to see each other. Considering that there are two parts during self-attention calculation, namely intra-relation selfattention which calculates the attention relationship between characters, and inter-relation self-attention which calculates the attention relationship between text instances. We consider whether two attention masks are needed to prevent information leakage. However, in fact, for intra-relation self-attention which calculates the attention relationship between characters, we do not need to use an attention mask, because, for any text instance (including the denoising part and the matching part), the attention calculation is within the text instance and does not interact with other text instances. So we only need to consider the design of interrelation self-attention, we call this attention mask that only needs to be used once as Single Attention Mask, and we devise the attention mask A = [ a ij ] ( g +2 ) n × ( g +2 ) n as follows:

$$a _ { i j } = \begin{cases} \begin{array} { c c } 1, & \text{if $j<g\times2n$ and $\lfloor \frac{i}{2n}\rfloor\neq\lfloor \frac{j}{2n}\rfloor$;} \\ 1, & \text{if $j<g\times2n$ and $i\geqg\times2n$;} \end{array} \end{cases} ( 7 )$$

where g and n represent the number of groups and the number of text instances per image, respectively. a ij = 1 means the i -th query is blind to the j -th query; if it's 0, they can see each other.

## 3.4. Training Losses

Compared to the matching part, the denoising part uses a slightly modified loss function and an additional background computation loss function (namely, the background part is calculated twice, which we briefly refer to as BCT). The rest of the loss functions are consistent with DeepSolo [48], including the Hungarian matching algorithm, focal loss, CTC loss [12], and L1 loss.

We use focal loss [24] to calculate the classification of text instances. In each set of denoising queries, the positive part represents positive samples, and the negative part represents negative samples, with the focal loss being used to compute the background loss for the first time. Therefore, for the τ -th query in the positive part of the denoising queries, the calculation of the focal loss for text instance classification is as follows:

$$\mathcal { L } _ { \text{cls} } ^ { ( \tau ) } & = - \, 1 _ { \{ \tau \in \text{Im} ( \varphi ) \} } \alpha ( 1 - \hat { b } ^ { ( \tau ) } ) ^ { \gamma } \log ( \hat { b } ^ { ( \tau ) } ) \\ & \quad - \, 1 _ { \{ \tau \notin \text{Im} ( \varphi ) \} } ( 1 - \alpha ) ( \hat { b } ^ { ( \tau ) } ) ^ { \gamma } \log ( 1 - \hat { b } ^ { ( \tau ) } ),$$

where 1 represents the indicator function, and Im ( φ ) denotes the image of the mapping φ . Concerning character classification, for the τ -th denoised query in the positive part, we employ the CTC loss:

$$\mathcal { L } _ { \text{text,pos} } ^ { ( \tau ) } = 1 _ { \{ \tau \in \text{Im} ( \varphi ) \} } C \text{TC} ( t ^ { ( \varphi ^ { - 1 } ( \tau ) ) }, \hat { t } ^ { ( \tau ) } ). \quad ( 9 )$$

The cross entropy loss for the κ -th denoised query in the negative part during the second background calculation:

$$\mathcal { L } _ { \text{text,neg} } ^ { ( \kappa ) } = \mathbb { 1 } _ { \{ \kappa \in \text{Im} ( \varphi ) \} } \text{CE} ( t ^ { ( \varphi ^ { - 1 } ( \kappa ) ) }, \hat { t } ^ { ( \kappa ) } ). \quad ( 1 0 )$$

Additionally, for the coordinate points of the center curve and boundaries in the positive part of the τ -th denoised query, we employ the L1 loss for the computation:

$$\mathcal { L } _ { \text{coord} } ^ { ( \tau ) } = \mathbf 1 _ { \{ \tau \in \text{Im} ( \varphi ) \} } \sum _ { n = 0 } ^ { N - 1 } \left \| p _ { n } ^ { ( \varphi ^ { - 1 } ( \tau ) ) } - \hat { p } _ { n } ^ { ( \tau ) } \right \|, \quad ( \mathbf 1 \mathbf 1 )$$

$$\mathcal { L } _ { \text{bd} } ^ { ( \tau ) } = 1 _ { \{ \tau \in \text{Im} ( \varphi ) \} } \sum _ { n = 0 } ^ { N - 1 } \left ( \left \| t o p _ { n } ^ { ( \varphi ^ { - 1 } ( \tau ) ) } - \hat { t o p } _ { n } ^ { ( \tau ) } \right \| \quad \text{(12)} \quad \text{emp} \\ + \left \| b o t _ { n } ^ { ( \varphi ^ { - 1 } ( \tau ) ) } - \hat { b o t } _ { n } ^ { ( \tau ) } \right \| \right ), \quad \text{pred} \quad \text{seTe}$$

where top refers to the top curves of the boundaries, and bot refers to the bottom curves of the boundaries. The negative part does not participate in the calculation of the part.

The loss function for the denoised queries consists of four aforementioned losses in the positive part and two aforementioned losses in negative part:

$$\mathcal { L } _ { \text{pos} } = \sum _ { \tau } \left ( \lambda _ { \text{cls} } \mathcal { L } _ { \text{cls} } ^ { ( \tau ) } + \lambda _ { \text{text,pos} } \mathcal { L } _ { \text{text,pos} } ^ { ( \tau ) } \quad \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \
 \quad _ { \text{$f_v=\vee\vee}$} } ^ { \quad \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \
 \end{cases}$$

+

λ

(

τ

)

(

τ

)

)

coord

L

coord

+

λ

bd

L

bd

,

$$\mathcal { L } _ { \text{neg} } = \sum _ { \kappa } \left ( \lambda _ { \text{cls} } \mathcal { L } _ { \text{cls} } ^ { ( \kappa ) } + \lambda _ { \text{text}, \text{neg} } \mathcal { L } _ { \text{text}, \text{neg} } ^ { ( \kappa ) } \right ), \quad ( 1 4 ) \quad 4. 3 \\ \mathcal { H } _ { \text{neg} }.$$

where λ cls , λ text , λ coord , λ bd are hyper-parameters to balance different tasks. The final loss function of the denoising part is:

$$\mathcal { L } _ { \mathrm d n } = \mathcal { L } _ { \text{pos} } + \mathcal { L } _ { \text{neg} }. \quad \quad \ \ ( 1 5 ) \quad \ \ s i$$

## 4. Experiment

We conduct comparisons with known Transformerbased approaches on various datasets, including TotalText, SCUT-CTW1500, ICDAR15, and InverseText which contain multi-directional scene text and arbitrary-shaped text instances. In addition, We choose publicly available datasets Synth150K, MLT17, IC13, and TextOCR as additional pre-training datasets.

## 4.1. Public Datasets

Total-Text is a widely used comprehensive scene text dataset introduced by [28], specifically designed for arbitrary text detection. It comprises 1255 training images and 300 testing images, containing horizontal, multidirectional, and arbitrary-shaped text instances.

SCUT-CTW1500 is another significant dataset for arbitrary-shaped text, published by [7]. It comprises 1500 images, consisting of 1000 training images and 500 testing images.

ICDAR2015 Incidental Text (ICDAR15) [16] includes 1000 training images and 500 testing images with quadrilateral text. It contains multi-directional text instances annotated with word-level quadrilateral annotations.

InverseText was manually annotated by [47] and includes 500 test images. Unlike the previous datasets used in detection tasks [2, 3], 40% of this dataset's text instances are inverse-like, specifically designed to address the lack of such texts in existing test datasets.

## 4.2. Implementation Details

All settings are based on the ResNet-50 backbone. We employ 6 layers of encoder and 6 layers of decoder, with a hidden dimension of 256 . For character classification, we predict 37 classes on the Total Text, ICDAR15 , and InverseText datasets, and 96 classes on the CTW1500 dataset. During training, we set the noise hyperparameters as follows: The probability λ of characters being flipped to other characters is set to 0 4 . . The learning rate scheduler utilizes an initial learning rate of 2 e -5 for the backbone and 2 e -4 for other parts. We train the DNTextSpotter for a total of 435 k steps, and the learning rate is reduced by a factor of 0 1 . at 375 k steps. We use AdamW as the optimizer and train our network with a batch size of 8 . The denoising part loss weights λ cls , λ coord , λ bd, λ text,pos , and λ text,neg are set to 1 0 . , 1 0 . , 0 5 . , 0 5 . , and 0 5 . . The focal loss parameters α and γ are set to 0 25 . and 2 0 . , respectively.

## 4.3. Comparison with State-of-the-Art Methods

For Arbitrarily-Shaped Scene Text Spotting: As previously indicated, the Total-Text and SCUT-CTW1500 datasets are specifically designed to emphasize text instances characterized by arbitrary shapes. In comparison to other methods on the Total-Text dataset (shown in Table 1), in the detection task, our approach is close to the current state-of-the-art method, ESTextSpotter, achieving 89 2% . . While its detection performance is slightly lower, it has a significant advantage in recognition performance. Without a lexicon ('None' results), outperforming ESTextSpotter by 3 7% . , reaching 84 5% . , and with a lexicon ('Full' results), exceeding it by 2 7% . , achieving 89 8% . . Compared to the state-of-the-art method in recognition performance, DeepSolo, is higher by 2 0% . and 1 1% . , respectively. On the CTW1500 dataset, both detection and recognition performances have reached the current state-of-the-art. The detection F1 score reached 90 2% . , surpassing the state-ofthe-art method ESTextSpotter by 0 2% . . Without a lexicon ('None' results), it exceeds ESTextSpotter by 2 1% . , and with a lexicon ('Full' results), it surpasses by 0 3% . , reaching 84 2% . . Compared to our baseline model, DeepSolo, there is a significant improvement in performance for detection and recognition, with increases in F1, 'None', and 'Full' by 0 9% 2 8% . , . , and 2 8% . , respectively. We use only ResNet-50 [13] as the backbone, and the results on various datasets reach state-of-the-art. When we switch the backbone to ViTAEv2-S, our performance also greatly exceeds that of DeepSolo using the same backbone.

For Arbitrarily-Oriented Scene Text Spotting: In the case of the multi-oriented benchmark ICDAR15, DNTextSpotter exhibits excellent performance when compared to other Transformer-based methods. As shown in Table 3, DNTextSpotter achieves results of 88 7% . , 84 3% . , and 79 9% . on the settings of 'S', 'W', and 'G', respec-

Table 1. Performances on Total-Text and CTW1500 with different backbone. E2E denotes the end-to-end spotting results. 'None' denotes lexicon-free. 'Full' denotes the inclusion of all words present in the test dataset. The top three scores are shown in bold red , blue , and green fonts. Additionally, results without TextOCR in pre-training are indicated with '*'.

|                      |                       | Total Text   | Total Text   | Total Text   | Total Text   | Total Text   | CTW1500   | CTW1500   | CTW1500   | CTW1500   | CTW1500   |        |
|----------------------|-----------------------|--------------|--------------|--------------|--------------|--------------|-----------|-----------|-----------|-----------|-----------|--------|
| Method               | Backbone              | Detection    | Detection    | Detection    | E2E          | E2E          | Detection | Detection | Detection | E2E       | E2E       | FPS    |
|                      |                       | P            | R            | F            | None         | Full         | P         | R         | F         | None      | Full      |        |
| TextDragon [10]      | VGG16                 | 85.6         | 75.7         | 80.3         | 48.8         | 74.8         | 84.5      | 82.8      | 83.6      | 39.7      | 72.4      | -      |
| SRSTS [44]           | ResNet-50             | 92.0         | 83.0         | 87.2         | 78.8         | 86.3         | -         | -         | -         | -         | -         | 18.7   |
| CharNet [45]         | ResNet-50-Hourglass57 | 88.6         | 81.8         | 84.6         | 63.6         | -            | -         | -         | -         | -         | -         | 1.2    |
| TextPerceptron [36]  | ResNet-50-FPN         | 88.8         | 81.8         | 85.2         | 69.7         | 78.3         | -         | -         | -         | -         | 57 . 0    | -      |
| Boundary [40]        | ResNet-50-FPN         | 88.9         | 85.0         | 87.0         | 65.0         | 76.1         | -         | -         | -         | -         | 46 . 1    | 73 . 0 |
| PGNet [41]           | ResNet-50-FPN         | 85.5         | 86.8         | 86.8         | 63.1         | -            | -         | -         | -         | -         | -         | 35.5   |
| ABCNet v2 [29]       | ResNet-50-FPN         | 90.2         | 84.1         | 87.0         | 70.4         | 78.1         | 85.6      | 83.8      | 84.7      | 57.5      | 77.2      | 10.0   |
| TPSNet [43]          | ResNet-50-FPN         | 90.2         | 86.8         | 88.5         | 78.5         | 84.1         | 88.7      | 86.3      | 87.5      | 60.5      | 80.1      | 14.3   |
| GLASS [37]           | ResNet-50-FPN         | 90.8         | 85.5         | 88.1         | 79.9         | 86.2         | -         | -         | -         | -         | -         | 3.0    |
| SwinTextSpotter [14] | Swin-T-FPN            | -            | -            | 88.0         | 74.3         | 84.1         | -         | -         | 88.0      | 51.8      | 77.0      | 2.9    |
| UNITS [17]           | Swin-B                | -            | -            | 89.8         | 78.7         | 86.0         | -         | -         | -         | -         | -         | -      |
| TESTR [52]           | ResNet-50             | 93.4         | 81.4         | 86.9         | 73.3         | 83.9         | 92.0      | 82.6      | 87.1      | 56.0      | 81.5      | 5.5    |
| TTS [18]             | ResNet-50             | -            | -            | -            | 78.2         | 86.3         | -         | -         | -         | -         | -         | -      |
| SPTS [35]            | ResNet-50             | -            | -            | -            | 74.2         | 82.4         | -         | -         | -         | 63.6      | 83.8      | 0.4    |
| ESTextSpotter [15]   | ResNet-50             | 92.0         | 88.1         | 90.0         | 80.8         | 87.1         | 91.5      | 88.6      | 90.0      | 64.9      | 83.9      | 4.3    |
| DeepSolo* [48]       | ResNet-50             | 93.1         | 82.1         | 87.3         | 79.7         | 87.0         | 92.5      | 86.3      | 89.3      | 64.2      | 81.4      | 17.0   |
| DeepSolo [48]        | ResNet-50             | 93.2         | 84.6         | 88.7         | 82.5         | 88.7         | -         | -         | -         | -         | -         | 17.0   |
| DeepSolo [48]        | VITAEv2-S             | 92.9         | 87.4         | 90.0         | 83.6         | 89.6         | -         | -         | -         | -         | -         | 10.0   |
| DNTextSpotter*(Ours) | ResNet-50             | 92.4         | 84.2         | 88.1         | 82.2         | 88.0         | 92.4      | 87.1      | 89.7      | 65.5      | 83.1      | 17.0   |
| DNTextSpotter(Ours)  | ResNet-50             | 91.5         | 87.0         | 89.2         | 84.5         | 89.8         | 93.5      | 87.1      | 90.2      | 67.0      | 84.2      | 17.0   |
| DNTextSpotter(Ours)  | VITAEv2-S             | 92.9         | 88.6         | 90.7         | 85.0         | 90.5         | 94.2      | 88.9      | 91.5      | 69.2      | 85.9      | 10.0   |

Table 2. Performance on Inverse-Text. The top three scores are shown in bold red , blue , and green fonts.

Table 3. Performance on ICDAR15. 'S,' 'W', and 'G' correspond to Strong, Weak, and Generic lexicon, respectively. The top three scores are shown in bold red , blue , and green fonts.

| Method                                 | E2E   | E2E   |
|----------------------------------------|-------|-------|
| Method                                 | None  | Full  |
| MaskTextSpotter v2 [22](ResNet-50-FPN) | 39.0  | 43.5  |
| ABCNet [27](ResNet-50-FPN)             | 22.2  | 34.3  |
| ABCNet v2 [29](ResNet-50-FPN)          | 34.5  | 47.4  |
| TESTR [52](ResNet-50)                  | 34.2  | 41.6  |
| SwinTextSpotter [14](Swin-T-FPN)       | 55.4  | 67.9  |
| SPTS [35](ResNet-50)                   | 38.3  | 46.2  |
| ESTextSpotter [15](ResNet-50)          | 51.2  | 55.1  |
| DeepSolo [48] (ResNet-50)              | 64.6  | 71.2  |
| DeepSolo [48] (ViTAEv2-S)              | 68.8  | 75.8  |
| DNTextSpotter(ResNet-50)               | 75.9  | 81.6  |
| DNTextSpotter(ViTAEv2-S)               | 78.1  | 83.8  |

tively. These results surpass SOTA method, DeepSolo, by 0 6% . in 'S', 0 4% . in 'W', and 0 4% . in 'G', respectively.

For

Inverse-like

Scene

Text

Spotting:

Besides

ES-

TextSpotter, whose weights were measured using the pub- licly available weights from the paper, all other results were

| Method                           | E2E   | E2E   | E2E   |
|----------------------------------|-------|-------|-------|
|                                  | S     | W     | G     |
| ABCNet v2 [29](ResNet-50-FPN)    | 82.7  | 78.5  | 73.0  |
| SwinTextSpotter [14](Swin-T-FPN) | 83.9  | 77.3  | 70.5  |
| TESTR [52](ResNet-50)            | 85.2  | 79.4  | 73.6  |
| SPTS [35](ResNet-50)             | 77.5  | 70.2  | 65.8  |
| ESTextSpotter [15](ResNet-50)    | 87.5  | 83.0  | 78.1  |
| DeepSolo [48](ResNet-50)         | 88.0  | 83.5  | 79.1  |
| DeepSolo [48](ViTAEv2-S)         | 88.1  | 83.9  | 79.5  |
| DNTextSpotter(ResNet-50)         | 88.7  | 84.3  | 79.9  |
| DNTextSpotter(ViTAEv2-S)         | 89.4  | 85.2  | 80.6  |

taken from the DeepSolo report. In the latest Inverse-Text dataset, our method achieves significant success. Compared to the current SOTA method, DeepSolo, our results without a lexicon ('None') surpassed it by 11 3% . , reaching 75 9% . , and with a lexicon ('Full'), we exceeded it by

Table 4. Comparison of DNTextSpotter results (black) and DeepSolo results (blue) at different Training Steps. The learning rate decays by a factor of 0.1 at 375K steps.

| Training   | InverseText   | InverseText   | TotalText   | TotalText   |
|------------|---------------|---------------|-------------|-------------|
| Steps      | None          | Full          | None        | Full        |
| 25K        | 27.9 (25.2)   | 38.2 (34.2)   | 52.5 (52.5) | 68.9 (65.3) |
| 50K        | 32.1 (29.5)   | 35.5 (35.3)   | 65.0 (60.0) | 74.2 (70.3) |
| 100K       | 40.9 (39.2)   | 49.5 (47.3)   | 67.9 (64.9) | 79.9 (77.8) |
| 200K       | 48.8 (46.5)   | 55.8 (52.3)   | 73.1 (71.9) | 82.9 (82.2) |
| 300K       | 54.5 (48.6)   | 61.8 (54.4)   | 74.0 (71.5) | 84.1 (82.0) |
| 375K       | 58.7 (50.6)   | 69.4 (55.1)   | 74.9 (73.9) | 83.4 (81.9) |
| 435K       | 67.9 (59.1)   | 74.4 (65.8)   | 79.2 (76.6) | 85.1 (84.0) |

10 4% . , achieving 81 6% . . We analyze why our method performed exceptionally well on this dataset. We believe this is due to the fact that datasets for Inverse-like texts present a more complex challenge than those for arbitrarily shaped texts. Unlike conventional texts that follow left-to-right, these texts are ordered from right to left and are flipped downward, making it challenging for the original model to learn this pattern. Denoising training, as an auxiliary task, with its relatively simpler nature, can more easily help the model learn these uncommon or complex forms of text.

Table 5. Ablation comparison of different noise scales λ in the experiment, where λ represents the probability of a character flipping to another character.

| λ   | Detection   | Detection   | Detection   | E2E   | E2E   |
|-----|-------------|-------------|-------------|-------|-------|
| λ   | P           | R           | F1          | None  | Full  |
| 0.8 | 91.8        | 85.1        | 88.2        | 82.9  | 88.8  |
| 0.6 | 92.2        | 85.6        | 88.7        | 83.4  | 89.2  |
| 0.4 | 91.5        | 87.0        | 89.2        | 84.5  | 89.8  |
| 0.2 | 92.9        | 84.4        | 88.4        | 84.0  | 89.6  |
| 0.0 | 91.2        | 84.8        | 87.8        | 82.7  | 88.6  |

## 4.4. Ablation Studies

Ablation experiments were conducted on the Total-Text dataset. Table 4 further demonstrates the convergence effects of the improved denoising training. Table 6 shows the impact of noise scale and mask probability. Table 7 shows the effects of adding noise to Bezier control points(BCP), using masked character sliding(MCS), and calculating an additional background loss(BCT).

(1) Effect of BCP: We investigate the addition of noise on the Bezier control points rather than directly on the sampling points of the Bezier center curve. Using BCP leads to a noticeable improvement in 'F1' results, by approximately 0 8% . . We think BCP can contribute positional prior

Table 6. Adjustment of mask probability, where 'mask' refers to converting repeated characters into the background.

| Mask Probability   |   0% |   25% |   50% |   75% |
|--------------------|------|-------|-------|-------|
| F1 (%)             | 88.7 |  88.6 |  89.2 |  88.9 |
| None (%)           | 83.1 |  83.7 |  84.5 |  82.9 |
| Full (%)           | 88.6 |  88.9 |  89.8 |  88.4 |

Table 7. Ablation comparison of the proposed components. 'DN' denotes whether denoising training is employed. 'BCP' refers to the addition of noise to B´zier control points rather than directly to e the sampling points on the B´ ezier curve's centerline. 'MCS' denotes the use of masked character sliding, while 'BCT' indicates the use of an additional background loss calculation, i.e., background calculation twice.

| DN   | BCP   | MCS   | BCT   | Detection   | Detection   | Detection   | E2E   | E2E   |
|------|-------|-------|-------|-------------|-------------|-------------|-------|-------|
|      |       |       |       | P           | R           | F1          | None  | Full  |
| ✗    | ✗     | ✗     | ✗     | 93.2        | 84.6        | 88.7        | 82.5  | 88.7  |
| ✓    | ✗     | ✗     | ✗     | 91.5        | 85.1        | 88.1        | 82.1  | 88.0  |
| ✓    | ✓     | ✗     | ✗     | 92.3        | 85.9        | 88.9        | 82.5  | 88.4  |
| ✓    | ✓     | ✓     | ✗     | 93.2        | 85.5        | 89.0        | 84.1  | 89.2  |
| ✓    | ✓     | ✓     | ✓     | 91.5        | 87.0        | 89.2        | 84.5  | 89.8  |

information to the noised positional queries, which is not achieved by directly adding noise to the sampling points on the centerline. Conversely, if noise is directly added to these sampling points, the denoising training segment would miss out on the benefits of the smooth Bezier curve's positional priors, thereby negatively impacting the training outcome.

(2) Effect of MCS: Generating queries directly without MCSleads to a significant decrease in performance because it forces the model to learn a fixed postional output order. Introducing the sliding operation means that the model no longer learns targets based on a fixed position, which promotes a one-to-one alignment between character and position. The experimental results also confirm the effectiveness of MCS, showing a 1 6% . increase in 'None' results.

(3) Effect of BCT: Employing additional background loss calculation techniques (BCT) leads to improvements in both F1 scores ( +0 2% . ) and 'None' results ( +0 4% . ). BCT is actually a reuse of the negative part. Originally, the focal loss was used solely to calculate the loss for binary classification between the foreground and background. Now, an additional cross-entropy loss requires that each character in the negative part undergo multi-class classification.

- (4) Effect of Noise Scale and Mask Probability: The ablation experiments presented in Table 5 and Table 6 indicate that choosing an appropriate noise ratio is crucial, as both excessive and insufficient noise can impact the results. We conduct ablation experiments on the hyperparameter λ for randomly flipping characters, with the experimental results

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00355v3-figure-b5299cd7-1aa7-4f81-a037-8534763d1248.jpg)

Figure 5. Several instance examples: rows display ESTextSpotter, DeepSolo, and DNTextSpotter (Ours) visualizations, respectively. In the recognition results, blue within parentheses represents correct recognition, while red denotes incorrect ones; outside the parentheses, ∅ signifies no detection or no recognition took place. Additional visual analysis is provided in the appendix.

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00355v3-figure-f85b8d03-337c-4211-8d09-c5bcac7ab186.jpg)

every 10k iterations, with every 10k iterations considered as one group. By comparing the differences in indices between the i -th group and the ( i + 1) -th group, we obtain the results for IS . We visualize the IS results as shown in Fig. 6. Additionally, the training set of the Total Text contains a total of 1255 training images, with an average of 7 04 . text instances per image, so the largest possible IS is 7 04 . × 2 = 14 08 . .

Figure 6. For the IS of TESTR, DeepSolo, and DNTextSpotter, we trained for 120k steps under the same settings, calculating the IS at every consecutive 10k step interval.

showing the performance when λ is set between 0 0 . and 0 8 . . Furthermore, we control the noise scale λ at 0.4 to conduct ablation experiments on the mask probability. The experimental results show that either excessive or insufficient noise can affect the model's performance.

## 4.5. Qualitative Analysis

Instability Measurement: We utilize the analysis method for quantifying the instability( IS ) of bipartite graph matching proposed by DN-DETR [19]. The calculation formula can be found in the appendix. For a training image in the TotalText dataset, we calculate the indices of N proposals

Visualization Comparisons: Fig. 5 illustrates the experimental results on InverseText. ESTextSpotter significantly struggles with the recognition of inverse-like text. Even for some texts, distortions and deformations occur during detection. Although DeepSolo has greatly improved the recognition of these inverse-like texts, it faces challenges in recognizing all the more dense texts. As for DNTextSpotter, we achieve good performance on most of the inverse-like texts, indicating that denoising training has an increasingly positive effect on more complex tasks. More detailed visualization results can be found in the appendix.

## 5. Conclusion

In this paper, we propose a novel denoising training method based on the attributes of scene text. Our research indicates that devising a denoising training method that aligns the positions and contents of characters is highly effective. Future research could focus on developing denoising training methods that further align with task-specific characteristics to improve model performance. We hope our approach offers valuable insights for other researchers.

## Appendix

## A. Detection Results on Inverse-Text

In addition to comparing the recognition results of 'None' and 'Full', we also supplement the comparison of detection results here. We compare some mainstream endto-end methods. DNTextSpotter employs the same data augmentation and the same additional pre-training datasets as DeepSolo, a mixture of Synth150K, MLT17, Total-Text, IC13, IC15, and TextOCR. After fine-tuning on the TotalText for 2 k iterations, DNTextSpotter directly applies these updated weights to assess performance on the Inverse-Text dataset. It consistently outperforms current state-of-the-art methods, as shown in Table 8, achieving 94.3% precision, 77.2% recall, and an F1-score of 84.9%.

Table 8. Detection Performance on InverseText. The top two scores are shown in bold red and blue fonts.

| Method                        | Detection   | Detection   | Detection   |
|-------------------------------|-------------|-------------|-------------|
|                               | P           | R           | F1          |
| ABCNet [27](ResNet-50-FPN)    | 85.1        | 68.5        | 75.9        |
| ABCNet v2 [29](ResNet-50-FPN) | 87.1        | 64.6        | 74.2        |
| TESTR [52](ResNet-50)         | 91.8        | 54.4        | 68.3        |
| ESTextSpotter [15](ResNet-50) | 78.7        | 71.4        | 74.9        |
| DeepSolo [48](ResNet-50)      | 93.9        | 63.8        | 76.0        |
| DeepSolo [48](ViTAEv2-S)      | 95.1        | 69.1        | 80.0        |
| DNTextSpotter(ResNet-50)      | 94.3        | 77.2        | 84.9        |
| DNTextSpotter(ViTAEv2-S)      | 95.4        | 79.2        | 86.4        |

## B. Details of the Instability Measurement

We analyze the instability of bipartite graph matching used by DN-DETR [19]. In the main text, we group every 10 k iterations as one group. We adopt this setting here as well. For a training image, we represent the predicted text instances from transformer decoders at the i -th group as P i = { P , P i 0 i 1 , ..., P i N -1 } , where N signifies the total count of detected text instances, and the M ground truth text instances are denoted as G = { G , G , G , ..., G 0 1 2 M -1 } . After bipartite matching, we generate a vector W i = { W ,W ,..., W i 0 i 1 i N -1 } for the i -th iteration to capture the matching outcomes, defined by:

$$W _ { n } ^ { i } = \begin{cases} \ m, & \text{$$

The stability for a single training image at iteration i is then determined by the variance between its W i and W i +1 , calculated as:

̸

$$\begin{array} {$$

Figure 7. We present an example of the attention mask when the number of the group is equal to 2. The values in the gray region are set to True to prevent information leakage from the denoising part to the matching part. The values in the orange and blue region are set to False, and the attention scores for this region are computed.

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00355v3-figure-222568fa-652e-4cd5-a479-9c78c243f56a.jpg)

Here, ✶ ( ) · stands for the indicator function, where ✶ ( z ) = 1 if z is true, and 0 otherwise. The overall stability for iteration i across the dataset is obtained by averaging these stability values for all images.

## C. Single Attention Mask

Wefurther present the attention mask in a graphical form to facilitate a better understanding for the readers. The attention mask A = [ a ij ] ( g +2 ) n × ( g +2 ) n is shown in the main text as follows:

$$\log _ { \substack { \iota \\ \iota } } a _ { i j$$

The visualization can be seen in Fig. 7.

## D. More Qualitative Results on Benchmarks

We provide more visualization results for the TotalText, CTW1500, ICDAR15, and InverseText datasets in Fig. 8, Fig. 9, Fig. 10, and Fig. 11. From these visualization results, it can be seen that we achieve advanced detection and recognition effects. On inverse-like texts, our spotting performance also does not show any decline.

## E. Limitation and Discussion

Although DNTextSpotter achieves quite good performance, there are still some limitations. The most significant is the excessive overhead during training. Compared to the original vector shape of ( bs, 100 25 256) , , ,

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00355v3-figure-192a3af7-0b9f-48ce-9c95-ff2cd23acbbf.jpg)

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00355v3-figure-0e806116-812e-4bb8-852a-e7febbef6a0f.jpg)

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00355v3-figure-ace10c4d-bff0-4bbb-9e9f-e1f1c66a179d.jpg)

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00355v3-figure-844484e1-f39d-4635-969b-1990a476ad8d.jpg)

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00355v3-figure-c9304b25-bd72-4cc5-b72e-419ebac4627d.jpg)

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00355v3-figure-f2ad8888-9e94-4524-a6bc-cc91f5afd305.jpg)

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00355v3-figure-0fb1054e-bb19-4520-a19d-1647fcf6cb20.jpg)

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00355v3-figure-62e4988c-d143-4e72-b4dc-c2d2a92b42dc.jpg)

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00355v3-figure-10898d72-2f4f-4873-bd8f-ff89f0140c3a.jpg)

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00355v3-figure-e96e5e79-73d5-4b4c-8e2c-56e11a26a193.jpg)

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00355v3-figure-7f3afad1-b70b-47b8-bc3b-f5fe72164ff3.jpg)

Figure 8. Qualitative results on Total Text.

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00355v3-figure-5ec67896-0d26-40d9-abde-f684a7846169.jpg)

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00355v3-figure-a54ed547-9ed8-4e5e-8f08-374109c4a11e.jpg)

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00355v3-figure-3af4f6c2-b245-435b-a971-7a01d3ec0233.jpg)

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00355v3-figure-2c53e0c6-7912-4601-b622-9863c14638cd.jpg)

Figure 9. Qualitative results on CTW1500.

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00355v3-figure-dbd4019f-8987-4801-83c3-0da1a1508249.jpg)

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00355v3-figure-b79d022e-2fb9-427d-8c7a-2a1d8f2aac72.jpg)

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00355v3-figure-fa60d1b2-bba5-4f35-83ca-f98776413108.jpg)

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00355v3-figure-875f30ce-d028-485f-84d9-3010b2763901.jpg)

Figure 10. Qualitative results on ICDAR15.

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00355v3-figure-b65912ab-2fa2-4591-88dc-e62ea7d47fe2.jpg)

Figure 11. Qualitative results on InverseText.

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00355v3-figure-06da48d2-9739-48b5-b794-6a135a061396.jpg)

during denoising training, the maximum shape can reach ( bs, 200 25 256) , , . Given that the computational complexity of the self-attention mechanism increases quadratically, the increased computational cost when the sequence length grows from 100 to 200 is non-negligible. DNTextSpotter was trained using 8 NVIDIA Tesla H800 GPUs, requiring approximately 26 hours of training time. Fortunately, the denoising training does not add any overhead during infer-

ence, making it a worthwhile method for actual deployment and application. Additionally, we have only applied the denoising training method to the evaluation of English scene text datasets and have not experimented with Chinese. We look forward to DNTextSpotter achieving similarly good results on Chinese datasets as well.

## References

- [1] Youngmin Baek, Seung Shin, Jeonghun Baek, Sungrae Park, Junyeop Lee, Daehyun Nam, and Hwalsuk Lee. Character region attention for text spotting. In Computer VisionECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XXIX 16 , pages 504521. Springer, 2020. 1, 3
- [2] Hengyue Bi, Canhui Xu, Cao Shi, Guozhu Liu, Yuteng Li, Honghong Zhang, and Jing Qu. Srrv: A novel document object detector based on spatial-related relation and vision. IEEE Transactions on Multimedia , 2022. 6
- [3] Hengyue Bi, Canhui Xu, Cao Shi, Guozhu Liu, Honghong Zhang, Yuteng Li, and Junyu Dong. Hgr-net: Hierarchical graph reasoning network for arbitrary shape scene text detection. IEEE Transactions on Image Processing , 2023. 6
- [4] Fred L. Bookstein. Principal warps: Thin-plate splines and the decomposition of deformations. IEEE Transactions on pattern analysis and machine intelligence , 11(6):567-585, 1989. 3
- [5] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-toend object detection with transformers. In European conference on computer vision , pages 213-229. Springer, 2020. 3
- [6] Xiaoxue Chen, Lianwen Jin, Yuanzhi Zhu, Canjie Luo, and Tianwei Wang. Text recognition in the wild: A survey. ACM Computing Surveys (CSUR) , 54(2):1-35, 2021. 1, 2
- [7] Chee-Kheng Ch'ng, Chee Seng Chan, and Cheng-Lin Liu. Total-text: toward orientation robustness in scene text detection. International Journal on Document Analysis and Recognition (IJDAR) , 23(1):31-52, 2020. 6
- [8] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929 , 2020. 3
- [9] Haoqi Fan, Bo Xiong, Karttikeya Mangalam, Yanghao Li, Zhicheng Yan, Jitendra Malik, and Christoph Feichtenhofer. Multiscale vision transformers. In Proceedings of the IEEE/CVF international conference on computer vision , pages 6824-6835, 2021. 3
- [10] Wei Feng, Wenhao He, Fei Yin, Xu-Yao Zhang, and ChengLin Liu. Textdragon: An end-to-end framework for arbitrary shaped text spotting. In Proceedings of the IEEE/CVF international conference on computer vision , pages 9076-9085, 2019. 3, 7
- [11] Ross Girshick. Fast r-cnn. In Proceedings of the IEEE international conference on computer vision , pages 1440-1448, 2015. 3
- [12] Alex Graves, Santiago Fern´ andez, Faustino Gomez, and J¨rgen Schmidhuber. u Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks. In ICML , 2006. 5
- [13] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 770-778, 2016. 6
- [14] Mingxin Huang, Yuliang Liu, Zhenghao Peng, Chongyu Liu, Dahua Lin, Shenggao Zhu, Nicholas Yuan, Kai Ding, and Lianwen Jin. Swintextspotter: Scene text spotting via better synergy between text detection and text recognition. In proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 4593-4603, 2022. 7
- [15] Mingxin Huang, Jiaxin Zhang, Dezhi Peng, Hao Lu, Can Huang, Yuliang Liu, Xiang Bai, and Lianwen Jin. Estextspotter: Towards better scene text spotting with explicit synergy in transformer. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 1949519505, 2023. 2, 3, 7, 10
- [16] Dimosthenis Karatzas, Lluis Gomez-Bigorda, Anguelos Nicolaou, Suman Ghosh, Andrew Bagdanov, Masakazu Iwamura, Jiri Matas, Lukas Neumann, Vijay Ramaseshan Chandrasekhar, Shijian Lu, et al. Icdar 2015 competition on robust reading. In 2015 13th international conference on document analysis and recognition (ICDAR) , pages 1156-1160. IEEE, 2015. 6
- [17] Taeho Kil, Seonghyeon Kim, Sukmin Seo, Yoonsik Kim, and Daehee Kim. Towards unified scene text spotting based on sequence generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 15223-15232, 2023. 3, 7
- [18] Yair Kittenplon, Inbal Lavi, Sharon Fogel, Yarin Bar, R Manmatha, and Pietro Perona. Towards weakly-supervised text spotting using a multi-task transformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 4604-4613, 2022. 2, 3, 7
- [19] Feng Li, Hao Zhang, Shilong Liu, Jian Guo, Lionel M Ni, and Lei Zhang. Dn-detr: Accelerate detr training by introducing query denoising. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 13619-13627, 2022. 2, 3, 9, 10
- [20] Hui Li, Peng Wang, and Chunhua Shen. Towards end-to-end text spotting with convolutional recurrent neural networks. In Proceedings of the IEEE international conference on computer vision , pages 5238-5246, 2017. 2
- [21] Yanghao Li, Chao-Yuan Wu, Haoqi Fan, Karttikeya Mangalam, Bo Xiong, Jitendra Malik, and Christoph Feichtenhofer. Mvitv2: Improved multiscale vision transformers for classification and detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 4804-4814, 2022. 3
- [22] Minghui Liao, Pengyuan Lyu, Minghang He, Cong Yao, Wenhao Wu, and Xiang Bai. Mask textspotter: An end-toend trainable neural network for spotting text with arbitrary shapes. PAMI , 43(2):532-548, 2021. 7
- [23] Minghui Liao, Guan Pang, Jing Huang, Tal Hassner, and Xiang Bai. Mask textspotter v3: Segmentation proposal net-
- work for robust scene text spotting. In Computer VisionECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XI 16 , pages 706-722. Springer, 2020. 1, 3
- [24] Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Doll´ ar. Focal loss for dense object detection. In ICCV , 2017. 5
- [25] Shilong Liu, Feng Li, Hao Zhang, Xiao Yang, Xianbiao Qi, Hang Su, Jun Zhu, and Lei Zhang. Dab-detr: Dynamic anchor boxes are better queries for detr. arXiv preprint arXiv:2201.12329 , 2022. 3
- [26] Xuebo Liu, Ding Liang, Shi Yan, Dagui Chen, Yu Qiao, and Junjie Yan. Fots: Fast oriented text spotting with a unified network. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 5676-5685, 2018. 1
- [27] Yuliang Liu, Hao Chen, Chunhua Shen, Tong He, Lianwen Jin, and Liangwei Wang. Abcnet: Real-time scene text spotting with adaptive bezier-curve network. In proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 9809-9818, 2020. 3, 7, 10
- [28] Yuliang Liu, Lianwen Jin, Shuaitao Zhang, Canjie Luo, and Sheng Zhang. Curved scene text detection via transverse and longitudinal sequence connection. Pattern Recognition , 90:337-345, 2019. 6
- [29] Yuliang Liu, Chunhua Shen, Lianwen Jin, Tong He, Peng Chen, Chongyu Liu, and Hao Chen. Abcnet v2: Adaptive bezier-curve network for real-time end-to-end text spotting. IEEE Transactions on Pattern Analysis and Machine Intelligence , 44(11):8048-8064, 2021. 1, 3, 7, 10
- [30] Yuliang Liu, Jiaxin Zhang, Dezhi Peng, Mingxin Huang, Xinyu Wang, Jingqun Tang, Can Huang, Dahua Lin, Chunhua Shen, Xiang Bai, et al. Spts v2: single-point scene text spotting. arXiv preprint arXiv:2301.01635 , 2023. 3
- [31] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF international conference on computer vision , pages 10012-10022, 2021. 3
- [32] Shangbang Long, Xin He, and Cong Yao. Scene text detection and recognition: The deep learning era. International Journal of Computer Vision , 129:161-184, 2021. 1, 2
- [33] Pengyuan Lyu, Minghui Liao, Cong Yao, Wenhao Wu, and Xiang Bai. Mask textspotter: An end-to-end trainable neural network for spotting text with arbitrary shapes. In Proceedings of the European conference on computer vision (ECCV) , pages 67-83, 2018. 3
- [34] Depu Meng, Xiaokang Chen, Zejia Fan, Gang Zeng, Houqiang Li, Yuhui Yuan, Lei Sun, and Jingdong Wang. Conditional detr for fast training convergence. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 3651-3660, 2021. 3
- [35] Dezhi Peng, Xinyu Wang, Yuliang Liu, Jiaxin Zhang, Mingxin Huang, Songxuan Lai, Jing Li, Shenggao Zhu, Dahua Lin, Chunhua Shen, et al. Spts: single-point text spotting. In Proceedings of the 30th ACM International Conference on Multimedia , pages 4272-4281, 2022. 7
- [36] Liang Qiao, Sanli Tang, Zhanzhan Cheng, Yunlu Xu, Yi Niu, Shiliang Pu, and Fei Wu. Text perceptron: Towards endto-end arbitrary-shaped text spotting. In Proceedings of the AAAI conference on artificial intelligence , volume 34, pages 11899-11907, 2020. 1, 3, 7
- [37] Roi Ronen, Shahar Tsiper, Oron Anschel, Inbal Lavi, Amir Markovitz, and R Manmatha. Glass: global to local attention for scene-text spotting. In ECCV , 2022. 7
- [38] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems , 30, 2017. 3
- [39] Zhaoyi Wan, Jielei Zhang, Liang Zhang, Jiebo Luo, and Cong Yao. On vocabulary reliance in scene text recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 11425-11434, 2020. 1
- [40] Hao Wang, Pu Lu, Hui Zhang, Mingkun Yang, Xiang Bai, Yongchao Xu, Mengchao He, Yongpan Wang, and Wenyu Liu. All you need is boundary: Toward arbitrary-shaped text spotting. In Proceedings of the AAAI conference on artificial intelligence , volume 34, pages 12160-12167, 2020. 1, 3, 7
- [41] Pengfei Wang, Chengquan Zhang, Fei Qi, Shanshan Liu, Xiaoqiang Zhang, Pengyuan Lyu, Junyu Han, Jingtuo Liu, Errui Ding, and Guangming Shi. Pgnet: Real-time arbitrarilyshaped text spotting with point gathering network. In AAAI , 2021. 7
- [42] Wenhai Wang, Enze Xie, Xiang Li, Xuebo Liu, Ding Liang, Zhibo Yang, Tong Lu, and Chunhua Shen. Pan++: Towards efficient and accurate end-to-end spotting of arbitrarilyshaped text. IEEE Transactions on Pattern Analysis and Machine Intelligence , 44(9):5349-5367, 2021. 1
- [43] Wei Wang, Yu Zhou, Jiahao Lv, Dayan Wu, Guoqing Zhao, Ning Jiang, and Weipinng Wang. Tpsnet: Reverse thinking of thin plate splines for arbitrary shape scene text representation. In Proceedings of the 30th ACM International Conference on Multimedia , pages 5014-5025, 2022. 7
- [44] Jingjing Wu, Pengyuan Lyu, Guangming Lu, Chengquan Zhang, Kun Yao, and Wenjie Pei. Decoupling recognition from detection: Single shot self-reliant scene text spotter. In ACM MM , 2022. 7
- [45] Linjie Xing, Zhi Tian, Weilin Huang, and Matthew R Scott. Convolutional character networks. In ICCV , 2019. 7
- [46] Yufei Xu, Qiming Zhang, Jing Zhang, and Dacheng Tao. Vitae: Vision transformer advanced by exploring intrinsic inductive bias. Advances in neural information processing systems , 34:28522-28535, 2021. 3
- [47] Maoyuan Ye, Jing Zhang, Shanshan Zhao, Juhua Liu, Bo Du, and Dacheng Tao. Dptext-detr: Towards better scene text detection with dynamic points in transformer. In AAAI , 2023. 6
- [48] Maoyuan Ye, Jing Zhang, Shanshan Zhao, Juhua Liu, Tongliang Liu, Bo Du, and Dacheng Tao. Deepsolo: Let transformer decoder with explicit points solo for text spotting. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 19348-19357, 2023. 2, 3, 5, 7, 10
- [49] Qixiang Ye and David Doermann. Text detection and recognition in imagery: A survey. IEEE transactions on pattern analysis and machine intelligence , 37(7):1480-1500, 2014. 1
- [50] Hao Zhang, Feng Li, Shilong Liu, Lei Zhang, Hang Su, Jun Zhu, Lionel M Ni, and Heung-Yeung Shum. Dino: Detr with improved denoising anchor boxes for end-to-end object detection. arXiv preprint arXiv:2203.03605 , 2022. 2, 3
- [51] Qiming Zhang, Yufei Xu, Jing Zhang, and Dacheng Tao. Vitaev2: Vision transformer advanced by exploring inductive bias for image recognition and beyond. International Journal of Computer Vision , pages 1-22, 2023. 3
- [52] Xiang Zhang, Yongwen Su, Subarna Tripathi, and Zhuowen Tu. Text spotting transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 9519-9528, 2022. 1, 3, 7, 10
- [53] Humen Zhong, Jun Tang, Wenhai Wang, Zhibo Yang, Cong Yao, and Tong Lu. Arts: Eliminating inconsistency between text detection and recognition with auto-rectification text spotter. arXiv preprint arXiv:2110.10405 , 2021. 3
- [54] Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai. Deformable detr: Deformable transformers for end-to-end object detection. arXiv preprint arXiv:2010.04159 , 2020. 1, 3
Yu Xie Qian Qiao Jun Gao Tianxiang Wu Jiaqing Fan Yue Zhang Jielei Zhang Huyang Sun
DNTextSpotter: Arbitrary-Shaped Scene Text Spotting via Improved Denoising Training
More and more end-to-end text spotting methods based on Transformer
architecture have demonstrated superior performance. These methods utilize a
bipartite graph matching algorithm to perform one-to-one optimal matching
between predicted objects and actual objects. However, the instability of
bipartite graph matching can lead to inconsistent optimization targets, thereby
affecting the training performance of the model. Existing literature applies
denoising training to solve the problem of bipartite graph matching instability
in object detection tasks. Unfortunately, this denoising training method cannot
be directly applied to text spotting tasks, as these tasks need to perform
irregular shape detection tasks and more complex text recognition tasks than
classification. To address this issue, we propose a novel denoising training
method (DNTextSpotter) for arbitrary-shaped text spotting. Specifically, we
decompose the queries of the denoising part into noised positional queries and
noised content queries. We use the four Bezier control points of the Bezier
center curve to generate the noised positional queries. For the noised content
queries, considering that the output of the text in a fixed positional order is
not conducive to aligning position with content, we employ a masked character
sliding method to initialize noised content queries, thereby assisting in the
alignment of text content and position. To improve the model's perception of
the background, we further utilize an additional loss function for background
characters classification in the denoising training part.Although DNTextSpotter
is conceptually simple, it outperforms the state-of-the-art methods on four
benchmarks (Total-Text, SCUT-CTW1500, ICDAR15, and Inverse-Text), especially
yielding an improvement of 11.3% against the best approach in Inverse-Text
dataset.