## From Attributes to Natural Language: A Survey and Foresight on Text-based Person Re-identification

Fanzhi Jiang, Su Yang , Mark W. Jones, Liumei Zhang ∗ ∗

Abstract -Text-based person re-identification (Re-ID) is a challenging topic in the field of complex multimodal analysis, its ultimate aim is to recognize specific pedestrians by scrutinizing attributes/natural language descriptions. Despite the wide range of applicable areas such as security surveillance, video retrieval, person tracking, and social media analytics, there is a notable absence of comprehensive reviews dedicated to summarizing the text-based person Re-ID from a technical perspective. To address this gap, we propose to introduce a taxonomy spanning Evaluation , Strategy , Architecture , and Optimization dimensions, providing a comprehensive survey of the text-based person Re-ID task. We start by laying the groundwork for textbased person Re-ID, elucidating fundamental concepts related to attribute/natural language-based identification. Then a thorough examination of existing benchmark datasets and metrics is presented. Subsequently, we further delve into prevalent feature extraction strategies employed in text-based person ReID research, followed by a concise summary of common network architectures within the domain. Prevalent loss functions utilized for model optimization and modality alignment in text-based person Re-ID are also scrutinized. To conclude, we offer a concise summary of our findings, pinpointing challenges in textbased person Re-ID. In response to these challenges, we outline potential avenues for future open-set text-based person Re-ID and present a baseline architecture for text-based pedestrian image generation-guided re-identification ( TBPGR ) .

Index Terms -Person Re-Identification, Text, Natural Language, Attributes, Diffusion Model.

## I. INTRODUCTION

T HE demands in public safety and the subsequently installed surveillance networks make the manual tracking and identifying individuals increasingly challenging. Automatic person re-identification (Re-ID) has emerged as a popular research area in computer vision to address this issue. Also known as person search [154], person Re-ID specifically refers to recognizing and tracking specific individuals using non-overlapping images and videos captured by cameras, determining whether a specific query person appears persistently or momentarily across different times and locations [6]. It finds important applications in scenarios like searching for characters in movies, locating missing children, and tracking criminals [126]. The source data collection for these applications is usually from CCTV footage on the street. Given the constraints of the equipment and the environment, a number of challenges arise. These challenges include lighting variations, occlusions, background changes, pose variations,

∗ Corresponding author

This paper was produced by the IEEE Publication Technology Group. They are in Piscataway, NJ.

Manuscript received April 20, 2024; revised August 16, 2024.

and differences in camera resolutions. A typical person reidentification system involves conducting image queries and searching for corresponding individuals in a gallery of images or a video pool. It involves extracting features directly from pedestrian appearance images and performing direct matching, then ranking against existing appearance images in the gallery. These scenarios assume that visual examples of individual identities are always available as queries. However, some special cases where the visualization samples of personal identity are not available, we perform retrieval based on textual descriptions only, which is called text-based person re-identification, as shown in Figure 1.

## A. Attribute-based Person Re-ID

Text-based person re-identification is a special form of person re-identification that, instead of using image data, relies on descriptions to reflect a person's appearance [75]. In the task of text-based person Re-ID, a natural language description and an image are provided as input, and the output is the identification of the person matching the description [20]. There has been a desire to perform high-level semantic person search using free-form natural language descriptions. However, early research initially started with the attribute-based person re-identification [27], [178], [181], [183]. Attributebased methods (Pedestrian Attributes Recognition, PAR) stem from the high correlation between attributes and pedestrian images. Pedestrian attributes such as gender, age, clothing type, clothing color, etc. are arranged and combined to make the pedestrian identity appear distinguishable [179]. Extracting attribute information from pedestrian images can be achieved by using pre-trained classifiers or attribute detectors. These classifiers or detectors can recognize and locate different attributes(body part) in the images. These extracted pedestrian attributes information needs to be encoded into a feature vectors pattern matching. This often involves transforming discrete attributes into continuous feature representations, common methods include one-hot encoding and embedding encoding [27]. During the recognition process, matching is conducted by comparing the attribute feature vectors of query images and gallery images. Common matching methods include computing distances or similarities between feature vectors (such as Euclidean distance, cosine similarity, etc.). Finally, based on the results of attribute matching, the final recognition result is determined through fusion of matching scores from different attributes or by adopting decision strategies. The attribute-based person Re-ID approach improves recognition accuracy and retrieval interpretability. This is because these

Fig. 1: Conceptual diagram of text-based person Re-ID. Given a textual description of a target person collected from a street witness, a monitor uses the model aimed at retrieving the corresponding person image from a given database of images collected from street CCTV.

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00096v1-figure-1ef6d8e1-62b8-4c85-8c7a-8a87da47b597.jpg)

attributes can be considered as high-level semantic information that is robust to viewpoint changes and different viewing conditions [182].

Previous work on attribute-based person retrieval has attempted to reduce modality gaps by aligning each person category and its corresponding images in a joint embedding space through modal adversarial training [7], [157], or by enhancing the expressive power of embedding vectors for person categories and images hierarchically [27]. While these pioneering studies reveal important but less explored methods for person retrieval, there remains significant room for further improvement. Firstly, due to their adversarial learning strategy, they exhibit instability and high computational cost during training [6], [60]. For other, they incur high inference costs due to the need for additional networks to match high-dimensional embedding vectors [27]. More importantly, these methods treat person categories as independent labels of person images and overlook their relationships, such as how many attributes differ between them, despite the potential rich supervisory signals for learning better representations of person categories and images. Dong et al. [30] address this issue by capturing rich information of two modalities through hierarchical embedding. However, their model entails high computational complexity as it computes high-dimensional embeddings and deploys additional networks for matching. Yin et al. [157] and Cai et al. [6] learn a joint embedding space where person categories and images are directly matched. To bridge the modality gap, their embedding space is trained in a modal adversarial manner; however, due to the nature of min-max optimization, this often leads to unstable and slow convergence. Furthermore, all these methods suffer from a limitation where person categories are treated as separate labels and the important relationships among them are overlooked. In the joint embedding space of the two modalities, the loss pulls images of the same person category closer to achieve modal alignment. [54] in-

Fig. 2: Multi-scale feature fusion: The figure shows the basic operations regarding the fusion of image and text multi-scale features in person re-identification.

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00096v1-figure-9e668204-5bdc-4e77-87b7-10d2481beaa7.jpg)

troduces a novel loss, Adaptive Semantic Margin Regularizer (ASMR), for learning cross-modal embeddings in the context of attribute-based person retrieval.

## B. Natural Language-based Person Re-ID

Attribute-based person retrieval [69], [168] provides a method for person re-identification based on free-form descriptions. However, there is an awareness that the leverage of attributes to describe a person's appearance could be limited in some circumstances. For instance, the PETA dataset [27] defines 61 binary and 4 multi-class person attributes, along with hundreds of phrases used to describe a person's appearance. In addition, even with a constrained set of attributes, labeling them for large-scale person image datasets is challenging and expensive. Using free-form natural language (NL) descriptions

may offer more flexibility and robustness compared to these predefined attributes.

Attribute-based person retrieval also has limitations in terms of the results robustness. For instance, shuffling and flipping some attributes while using a basic fact set for retrieval can lead to significant performance degradation of the results. Compared to manually assigned sets of attributes, textual descriptions can be seen as an unstructured form of annotation. In general, most attributes fall into the category of noun phrases, and natural language includes higher-level semantic words and complex events than the attributes. In principle, natural language processing (NLP) techniques can facilitate all attributes with any possible words and lengths, capturing unique details through natural language descriptions. Additionally, descriptions can include indicators of certainty or fuzziness to retain as much information as possible, potentially allow sets of multi-class attribute assignments. Both attributebased and NLP-based person re-identification are referred to as text-based person re-identification in this paper.

Li et al. [73] introduced the first dataset for NL-based person Re-ID, evaluated and compared various possible models, and proposed an RNN with a gated neural attention mechanism. Jing et al. [59] proposed to focus on mining fine-grained local parts and performing fine-grained visual-textual matching. These methods can learn distinctive part representations and achieve 81.23% accuracy, but they are troubled by the problem of ambiguous embeddings. Image regions and words are correlated at different semantic granularities [136], [168]. The discriminative parts of a person usually appear with different granularities, which was overlooked in the previous methods and ambiguated embeddings. To alleviate this issue, words (based on NL) as guidance have been integrated into the related image regions. However, irrelevant words may mislead the model and even exacerbate the ambiguous embeddings problem.

Cross-domain embeddings technique is one of the main trends in the Re-ID community. Work proposed in [16] added supplementary annotations to extract meaningful image regions independently, these regions, however, are not always available in the real-world scenes. Work in [69] proposed to learn cross-modal embeddings to improve matching results using a shared attention mechanism. [161] designed a cross-modal projection matching loss and a cross-modal projection classification loss for learning discriminative imagetext embeddings. [148] proposed a matrix factorization-based dictionary learning algorithm to eliminate the impact of style and pedestrian pose information on cross-domain Re-ID. [17] proposed a patch-word matching model and designed an adaptive threshold mechanism in the model, it attempted to establish global and local image-language associations to enhance the semantic consistency between local visual and language features.

Deep adversarial graph attention convolutional network was also proposed to address the problem of text-based person re-identification [81], [102]. Niu et al. [94]proposed a Multilevel Image-Text Alignment (MIA) model, which introduced cross-modal adaptive attention in three different granularities, integrating these three similarities to comprehensively judge the affinity of image-sentence pairs and better distinguish interested pedestrians. Zhu et al. [173]achieved an environmentperson separation (DSSL) method under the mutual exclusion constraint, and also annotated another dataset for person search using language called Real Scene Text-based Person Re-ID (RSTPReID). Wu et al. [140] designed a twin task, image coloring and text completion, and explicitly built fine-grained cross-modal associations bidirectionally based on color reasoning, their LapsCore method achieved 63.4% rank@1 retrieval accuracy on the CUHK-PEDES dataset. [92] proposed a novel cross-modal matching method named Cross-modal Co-occurrent Attribute Alignment (C2A2), which can better handle noise and significantly improve retrieval performance in person search by language. Relying on matrix factorization to construct visual and textual attribute dictionaries and crossmodal alignment using denoising reconstruction features addresses noise generated by extraneous elements of pedestrians. Zheng et al. [167] made the first attempt to perform NL-based person search within unsegmented complete images.

## C. Person Re-Identification previous survey

In order to provide researchers with a comprehensive understanding of the current status and research directions in the domain of text-based person re-identification, we conducted an in-depth investigation of text-based person re-identification methodologies and synthesized recent research achievements. Prior to this effort, several researchers have conducted reviews in the field of person re-identification [4], [9], [50], [53], [62], [89], [90], [98], [108], [116], [119], [126], [134], [141], [146], [149], [159], [169], [176] and we summarize the primary contributions of these studies in the Table III. Among them, certain surveys [4], [9] introduced previous challenges in the field of person Re-ID. [62], [90], [98], [134], [141] summarized deep learning based pedestrian re-recognition methods. [108], [119] investigated the available image/video-based person ReID datasets. [159], [170] surveyed image-based or videobased pedestrian re-recognition systems. [61], [90], [154], [176] proposed metric learning based classification method for pedestrian re-recognition. [53], [90], [154] presented a pedestrian re-recognition taxonomy based on representation learning. Huang et al. [50] provided a visible-infrared crossmodal categorization of pedestrian re-identification studies. Wang et al. [126] reviewed cross-modality based person ReID research from four aspects, including sketch, text, low resolution (LR) and infrared (IR), as Figure 6 is introduced later.

However, these surveys exhibit areas for improvement, such as lacking a systematic categorization and analysis of textbased deep learning methods in person re-identification, and omitting a comprehensive discussion of text-based person reidentification. Therefore, in comparison to the aforementioned reviews, this paper delves into an in-depth analysis of recent research in text-based person re-identification from dimensions including datasets, strategies, architectures, and optimizations. We comprehensively review existing deep learning-based approaches and discuss their strengths and limitations.

TABLE I: The table summarizes the available public datasets for attribute-based and NL-based person Re-ID. The datasets are summarized in terms of dimensions such as dataset, number of pedestrians, attributes/description, and data source, respectively.

|                        | Dataset                                                | Pedestrians                   | Attribute/Description   | Attribute/Description   | Source                                        |
|------------------------|--------------------------------------------------------|-------------------------------|-------------------------|-------------------------|-----------------------------------------------|
|                        |                                                        |                               | Binary                  | Multi-class             | Outdoor/Indoor                                |
| Attribute-based        | PETA[9] RAP[88] RAP V2.0[89] PA-100K[90] UAV-Human[87] | 19000 41585 84928 32668 34183 | 61 69 7                 | 4 3 3 / 1 /             | Indoor Indoor Outdoor Outdoor Outdoor Outdoor |
|                        | CUHK-PEDES[4]                                          | 34054                         | 2                       | > 23                    | Outdoor+Indoor                                |
|                        |                                                        |                               | 69                      |                         |                                               |
|                        |                                                        | 100000                        | 26                      |                         |                                               |
|                        | Market1501-Attribute[8]                                |                               | 26                      |                         |                                               |
|                        | DukeMTMC-Attribute[8]                                  |                               | 23                      |                         |                                               |
|                        |                                                        | 22263                         |                         |                         | Outdoor                                       |
| Natural Language-based |                                                        |                               | Sentences *             | / words                 |                                               |
| Natural Language-based | RSTPReid[24]                                           | 20505                         | 2                       | > 25 . 8                | Outdoor+Indoor                                |
| Natural Language-based | ICFG-PEDES[65]                                         | 54522                         | 3                       | > 37 . 2                | Outdoor+Indoor                                |

## II. EVALUATION

The evaluation of text-based person Re-ID tasks, typically involves publicly available benchmark datasets and common performance metrics. This provides a standardized testing platform, enabling a quantified and objective comparative assessment of the effectiveness of different experimental algorithms.

there are three large publicly available datasets, CUHKPEDES, RSTPReid, and ICFG-PEDES. The details of these datasets will be described.

## A. Datasets

1) attribute-based Person Re-ID Datasets: Attribute-based Person Re-identification (Re-ID) often involves the utilization of semantic attributes that describe appearance characteristics of individuals, such as gender, age, clothing type, clothing texture, and clothing color [114]. However, these attributes frequently encompass low-level semantic attributes that can be directly associated with image regions, such as clothing type, clothing texture, and clothing color, as well as highlevel semantic attributes that directly correspond to people, such as gender and age. By leveraging these attributes, the negative impact of variations in viewpoint and appearance on person re-identification can be mitigated, resulting in more reliable recognition and tracking performance [54]. To facilitate research in Attribute-based Person Re-ID, researchers have initiated the construction of several datasets specifically tailored for this purpose. Notable publicly available datasets in this domain include PETA [27], UAV-Human [80], RAP [77], RAP2.0 [78], PA-100K [84], Market1501-Attribute [181], and DukeMTMC-Attribute [181].

These datasets typically consist of a substantial collection of images capturing individuals from surveillance cameras, each accompanied by attribute labels. The dataset creation process generally involves selecting a subset of images from existing Person Re-ID datasets. Subsequently, attribute labels are assigned to each image either manually or through automated annotation. Ultimately, the annotated images along with their corresponding attribute labels are organized into a unified dataset, which serves as a resource for researchers in the field, as shown in Table I.

2) Natural Language-based Person Re-ID Datasets: Publicly available NL-based person re-identification datasets have significant practical implications for benchmarking, algorithm improvement, security and surveillance applications, privacy protection, and advancing multimodal research. Currently, a) CUHK-PEDES: CUHK-PEDES is the first description dataset introduced by The Chinese University of Hong Kong [75], designed for training and evaluating person description and image retrieval tasks in computer vision. The dataset was collected by aggregating person images from five existing and well-known pedestrian re-identification evaluation datasets, namely CUHK03 [180], Market-1501 [169], SSM [142], VIPER [44], CUHK01 [67], and subsequently annotated with descriptive text by crowdsourced workers from Amazon Mechanical Turk (AMT). It consists of over 40,206 images with 80,412 person descriptions, each containing at least 23 words, across 13,003 unique person identities. The training set includes 34,054 images, 11,003 unique persons, and 68,108 textual descriptions. The validation set contains 3,078 images, 1,000 unique persons, and 6,158 textual descriptions, while the test set comprises 3,074 images, 1,000 unique persons, and 6,156 textual descriptions. The individuals in the dataset were captured in street and public places in the Hong Kong area. CUHK-PEDES provides diverse person descriptions encompassing appearance, actions, postures, and interactions, with each person description linked to the corresponding image ID. The release of the CUHK-PEDES dataset promotes interdisciplinary research between computer vision and natural language processing, providing researchers in the fields of person description and image retrieval with the first benchmark dataset.

b) RSTPReid: The RSTPReid dataset is a pedestrian re-identification dataset introduced by Nanjing University of Science and Technology [173], designed for training and evaluating pedestrian re-identification tasks in computer vision. To address the issue in the CUHK-PEDES dataset where each specific pedestrian is captured by the same camera under the same time-space conditions, which does not reflect realworld scenarios, the authors constructed the Real Scenarios Text-based Person Reidentification (RSTPReid) dataset based on MSMT17 [138]. The RSTPReid dataset contains 20,505 images of 4,101 individuals captured by 15 independent cameras from different viewpoints, lighting conditions, locations, and weather conditions. Each individual has 5 corresponding images taken by different cameras, and each image is accompanied by 2 text descriptions with no fewer than 23

words. The training set includes 3,701 identities, the validation set has 200 identities, and the test set also consists of 200 identities. The dataset includes a relatively small number of pedestrian identities but covers both indoor and outdoor scenarios. Each pedestrian appears in multiple images with different shooting angles and lighting conditions, making this dataset more challenging and realistic.

c) ICFG-PEDES: The Identity-Centric and Fine-Grained Person Description Dataset (ICFG-PEDES) is a person description dataset introduced by South China University of Technology [29], which also serves as a new benchmark dataset for research in person description and image retrieval domains. Similar to CUHK-PEDES, this dataset contains a large number of person descriptions paired with corresponding image IDs. ICFG-PEDES consists of 54,522 pedestrian images of 4,102 distinct identities, all collected from the MSMT17 database [138]. The training set comprises 34,674 image-text pairs of 3,102 pedestrians, while the test set contains 19,848 image-text pairs of 1,000 pedestrians. Each image is associated with only one text description, with an average of 37.2 words per description. Compared to the CUHK-PEDES dataset, ICFG-PEDES emphasizes more on fine-grained person descriptions and reduces some irrelevant action and background information. Additionally, it addresses the issue of consistent backgrounds in the former dataset by emphasizing more on appearance variations. Consequently, the ICFG-PEDES dataset can be employed for more challenging image retrieval and person description tasks.

## B. Metrics

1) Mean Average Precision(mAP): For each query i, we define a precision P i ( j ) as the proportion of correct matches in the top j matches, and then for each positive instance, we calculate the average of the proportions of all positive instances before this one, which is the Average Precision AP i :

$$A P _ { i } = \frac { 1 } { M _ { i } } \sum _ { j = 1 } ^ { M _ { i } } P _ { i } ( j ) \cdot I ( r a n k _ { i } = j ). \quad \ \ ( 1 ) \quad \text{$person$} \\ \quad \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \$$

Where M i is the number of positive instances for query i, and ∑ is the summation over all positive instances. The symbol I ( rank i = j ) usually denotes an indicator function. The indicator function takes the value of 1 under certain conditions, otherwise it takes the value of 0. Specifically:

$$I ( \text{rank} _ { i } = j )$$

indicates that the indicator function I ( rank i = j ) takes the value of 1 when the first i query's first j result is a correct match, otherwise it takes the value of 0. In other words, it determines whether or not there is a correct match in the ranked position of j . Then, mAP is the average of the Average Precision of all queries:

$$\ m A P = \frac { 1 } { N } \sum _ { i = 1 } ^ { N } A P _ { i }. \quad \ \ \ \ \ ( 2 ) \quad \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \$$

Where N is the total number of queries, and ∑ is the summation over all queries.

2) Rank-N Accuracy: For each query i, we define an indicator function I ( rank i ≤ n ) , which is 1 if the correct match is within the top n matches, and 0 otherwise. Then, Rank-n accuracy is the average of this function:

$$R a n k - n = \frac { 1 } { N } \sum _ { i = 1 } ^ { N } I ( r a n k _ { i } \leq n ). \quad \quad ( 3 )$$

Where N is the total number of queries, and ∑ is the summation over all queries.

## III. STRATEGY

In the task of text-based person re-identification, image data is typically captured through CCTV in open environments [45]. This data not only contains key descriptive features of individuals but also includes a significant amount of background noise. Early works [17], [75], [148], [161] typically directly extract global image text features for simple crossmodal matching. However, due to the uniqueness of the person re-identification task, which differs from existing image-text retrieval [60], [167], the categories to be retrieved all belong to highly similar individuals. Identification based solely on distinguishing features such as clothing and posture increases the challenges of the text-based person re-identification task.

## A. Stripe Segmentation

Some work suggests that various parts of the human body are evenly arranged in images, and Stripe Segmentation can be employed as a guiding model to achieve multimodal local alignment supervision [57], [94]. Stripe Segmentation is a commonly used technique in pedestrian re-identification, aiming to mitigate the impact of pose, occlusion, and lighting variations on re-identification performance within pedestrian images [94], [29]. The fundamental idea is to partition pedestrian images into multiple vertical stripes, also referred to as person parts. Each stripe can be considered a subregion of the pedestrian image (head, upper body, lower body, foot). The advantage of this approach lies in dividing pedestrian images into multiple local regions, effectively capturing local features of pedestrians to enhance recognition accuracy [79]. For instance, local texture, color, and shape features can be extracted within each stripe, facilitating better differentiation between different parts of pedestrians. Stripes of different heights can correspond to different scales of pedestrian parts, providing richer scale information to address scale variation issues in pedestrian images. Ding et al. [29] proposed a Semantic Self-Alignment Network (SSAN) capable of automatically extracting part-level textual features for corresponding visual regions. They designed a multi-view non-local network to capture relationships between body parts, thereby establishing better correspondences between body parts and noun phrases. Li et al [79] proposed to vertically divide a character image into multiple regions using two different approaches, including overlapping slices and key-point-based slices. Niu et al. [94] also emphasized local-local alignment, employing Stripe Segmentation in the proposed Bidirectional Fine-Grained Matching (BFM) module to match visual human

body parts with noun phrases. However, this strategy is fragile and sensitive to conditional changes. For example, the regions of image samples do not always contain a complete human body, and at times, the human head does not consistently appear in the first stripe, significantly impacting the robustness of existing methods.

## B. Multi-scale Fusion

Due to the small inter-class variance in both pedestrian images and descriptions, comprehensive information is required for pedestrian retrieval to coordinate visual and textual clues across all scales. Subsequent to the introduction of local feature extraction strategies other than stripe segmentation, the utilization of fused features is referred to as multi-scale feature fusion [130], as illustrated in Figure 3. Image Structure Graph Network (A-GANet) [81] employs residual modules for visual representation extraction and utilizes designed graph attention convolutional layers for structured high-level visual semantic feature extraction. Specifically, it captures relationship features of different pedestrian parts through graph convolutional networks. Wang et al. [136] proposed the Visual Text Attribute Alignment model (ViTAA), which learns to decompose the person's feature space into subspaces corresponding to attributes using an optically assisted attribute segmentation layer. On the image side, a segmentation layer is employed to divide pedestrian images into full body, head, upper clothes, Lower clothes, shoes, and bag parts. On the text side, corresponding phrases of different segmented parts are extracted.

Ji et al. [57] introduced an Asymmetric Cross-Scale Alignment (ACSA) method. Global text representation and local phrase representation are employed for text extraction. The global representation is divided into four regions as local visual representations, namely Head, Upper body, Lower body, and foot. These features are used afterwards for feature concatenation and computation. The partition strategy does not involve additional computational costs but can better retain prominent body parts for fine-grained matching. Wang et al. [135] proposed a novel Multi-Granularity Embedding Learning (MGEL) model. It generates multi-granularity embeddings of partial human bodies from coarse to fine by revisiting person images at different spatial scales. Chen et al. [20] adopted a multi-branch representation in the learning path, enabling text-adaptive matching of corresponding visual local representations, thereby aligning text and visual local representations. Additionally, a multi-stage cross-modal matching strategy was proposed to eliminate modality gaps from low-level, local, and global features, gradually narrowing the feature gap between image and text domains.

Niu et al. [94] presented a Multi-Granularity Image-Text Alignment (MIA) framework to address cross-modal finegrained issues. It consists of three modules corresponding to three granularities. The Global Contrast (GC) module is used for global-global alignment. The Relation-Guided Global-Local Alignment (RGA) module filters global-local relationships. The Bidirectional Fine-Grained Matching (BFM) module aligns local to local based on trained fine-grained local components. Zheng et al. [168] proposed a Hierarchical

Adaptive Matching model to learn subtle feature representations from three different granularities (i.e., word-level, phrase-level, and sentence-level) for fine-grained image-text retrieval tasks. Wang et al. [120] addressed the issue that existing methods typically learn similar mappings between local parts of images and text or embed entire images and text into a unified embedding space, resulting in a problem of local-global correlation. They designed a Divide and Merge Embedding (DME) learning framework for text-based person search. It models the relationship between local parts and global embedding, merging local details into the global embedding. Li et al. [76] constructed features for text and person image matching, different from other methods. The latter acquires global features by aggregating aligned local features, leading to misalignment between text and visual features. They proposed placing features with the same semantics in the same spatial position to construct the final features for person-text-image matching, achieving semantic consistency and interpretability of global features.

Yan et al. [151] proposed an Implicit Local Alignment module, adaptively aggregating image and text features into a set of modality-shared semantic topic centers, implicitly learning local fine-grained correspondence between images and text without additional supervision information and complex cross-modal interaction. Additionally, global alignment is introduced as a supplement to the local perspective. Gao et al. [41] introduced a method called Non-Local Alignment on Full Scale (NAFS), capable of adaptively aligning image and text features at all scales. First, a novel ladder network structure is proposed to extract full-size image features with better locality. Secondly, a language model, Bidirectional Encoder Representations from Transformers (BERT), with local constraint attention is proposed to obtain description representations at different scales. Then, instead of aligning features separately at each scale, a novel context non-local attention mechanism is applied to simultaneously discover potential alignment at all scales.

In summary, applying target detection or additional branch networks to detect significant regions and then extract local features for obtaining multi-scale features has certain accuracy advantages. However, due to the increased external network overhead, these methods often result in higher computational costs.

## C. Attention Mechanism

Cross-modal alignment is inherently challenging in achieving fine-grained matching between text and images. In addition to implicit alignment strategies achieved through the automatic fusion of multi-scale features, another key approach involves the use of explicit alignment strategies. For instance, employing attention mechanisms enhances the capability of feature extraction by capturing discriminative feature representations related to both language descriptions and visual appearances. This aids in intuitively understanding and controlling the alignment process. Attention models can establish correspondences between body parts and words [29], [35], [69], [75], [135], [148], [164], offering the advantage of not relying on

external cues for aligning feature embeddings. These attention strategies can be categorized into spatial attention, channel attention, mixed attention, non-local attention, and positional attention.

Liu et al. [135] proposed an attention-based deep neural network capable of capturing multiple attention features from low-level to semantic layers to learn comprehensive features for fine-grained pedestrian representation. Wang et al. [82] utilized a multi-head self-attention module to extract embeddings for different granularity portions of the text stream, employing adaptive filtering to remove interference and obtain fine-grained features. Lee et al. [63] introduced a Stacked Cross Attention Mechanism (SCAN) that compares attention information for text and image in two stages, determining the importance of image regions and words to achieve potential alignment between the two modalities. Spatial attention-based methods often focus solely on local discriminative features of pedestrians, neglecting the impact of feature diversity on pedestrian retrieval.

Li et al. [68] proposed a cubic attention convolutional neural network that combines spatial and channel attention to maximize complementary information from different scale attention features, addressing the challenges of cross-modal alignment in pedestrian re-identification. Wang et al. [120] designed a Feature Division Network (FDN) that embeds inputs into K locally guided semantic representations using self-attention, each representing different parts of a person. They then introduced a Correlation-based Subspace Projection (RSP) method to merge different local representations into a compact global embedding. Yan et al. [151] presented an Efficient Joint Information and Semantic Alignment Network (ISANet) for text-based person search. Specifically, they designed an imagespecific information suppression module that suppresses image background and environmental factors through relationshipguided localization and channel attention filtering, effectively alleviating information inequality and achieving information alignment between text and image.

Considering the indistinguishability of features extracted from first-order attention (e.g., spatial and channel attention) in complex camera views and pose-changing scenarios, Farooq et al. [35] applied non-local attention directly after calculating interactions between text features to model long-term dependencies between different phrases in the text. Inspired by self-attention, Gao et al. [41] proposed a Context NonLocal Attention to enable cross-modal features to align with each other in a coarse-to-fine manner based on their semantics, rather than relying solely on predefined and fixed rules (e.g., local-local, global-global). Wang et al. [130] introduced a Partbased Multi-scale Attention Network (PMAN), consisting of a dual-path feature extraction framework with attention-based branches for a multi-scale cross-modal matching module. This module extracts visual semantic features from different scales and matches them with text features.

Addressing concerns that utilizing soft attention mechanisms to infer semantic alignment between image regions and corresponding words in a sentence may lead to the fusion of unrelated multimodal features and result in redundant matching, Wang et al. [123] proposed an IMG-Net model. This model combines intra-modal self-attention and cross-modal hard region attention with a fine-grained model to extract multi-granularity semantic information. Zheng et al. [168] introduced a Hierarchical Gumbel Hard Attention Module, using the Gumbel top-k reparameterization algorithm as a low-variance, unbiased gradient estimator to select strongly semantically related regions for all regions of the image and corresponding words or phrases, enhancing precise alignment and similarity calculation results.

While these techniques often bring about noticeably superior retrieval performance, they frequently come at the cost of efficiency. Specifically, for M galleries and N queries, their complexity is O M ( + N ) . In contrast, the complexity of attention-based cross-modal approaches rises to O MN ( ) due to pairwise inputs. To address this issue, Yang et al. [150] initially employ cross-modal non-attention features to quickly identify candidates and then deploy attention-based modules to refine the final ranking scores.

## D. External Auxiliary

For Text-based Person Re-Identification (Re-ID) tasks, several auxiliary strategies such as image segmentation, random mask, human body keypoints, attribute prediction, clustering analysis, and color extraction can be employed to extract advanced semantics, thereby improving cross-modal retrieval performance. [101] proposed a dual-part simultaneous alignment representation scheme using attention mechanisms to capture distinguishing information beyond body parts by leveraging complementary information from accurate human parts and noisy parts to update representations.

To leverage multi-level visual content, Jing et al. [60] introduced a Pose-guided Multi-granularity Attention Network (PMA). They initially proposed a Coarse Alignment network (CA) that uses similarity-based attention to select relevant image regions for global description. Pose information can be utilized to learn potential semantic alignments between visual body parts and textual noun phrases. Shu et al. [107] introduced Bidirectional Mask Modeling (BMM) without the need for additional manual labeling. They applied random masks to images and corresponding text keywords, forcing the model to explore more useful matching clues. This approach increases data diversity and enhances model generalization.

Wang et al. [136] proposed a novel Visual-Textual Attribute Alignment model (ViTAA). For semantic feature extraction, segmentation labels are used to drive the learning of attributeaware features from input images. Aggarwal et al. [1] introduced a method for text-based person search by learning attribute-driven spatial and class-driven spatial information. The goal is to create semantically retained embeddings through an additional task of attribute prediction. Suo et al. [112] proposed a novel Simple Robust Correlated Filtering (SRCF) framework, which effectively extracts key clues and adaptively aligns multimodal features by constructing a set of universal semantic templates (filters). This framework differs from previous attention-based methods, focusing on computing similarities between templates and inputs. Two types of filtering modules (denoising filter and dictionary filter) were designed to extract key features and establish multimodal mappings.

Zhao et al. [165] introduced a Cross-Modal Mutual Training (CMMT) framework. Specifically, to alleviate intra-class variation, clustering methods generate pseudo-labels for visual and text instances. To further refine clustering results, CMMT provides a mutual pseudo-label refinement module, which utilizes clustering results in one modality to refine clustering results in another modality constrained by text-image pairwise relationships. Chen et al. [18] integrated information theory and adversarial learning into an end-to-end framework, exploring information theory to reduce heterogeneity gaps in crossmodal retrieval. This method is beneficial for building a shared space to further learn commonalities between cross-modal features, applicable to video-text matching. Additionally, regularization terms based on KL-divergence and temperature scaling were introduced to address the issue of data imbalance.

Li et al. [70] proposed a novel Joint Label and Feature Alignment Framework (TFAF) to gradually reduce inter-modal and intra-class gaps. Firstly, a dual-path feature learning network is constructed to extract features and align them to reduce inter-modal gaps. Secondly, a text generation module is designed to generate label sequences using visual features, followed by label alignment to reduce intra-class gaps. Finally, a fusion interaction module is introduced, utilizing a multistage feature fusion strategy to further eliminate modality heterogeneity. Chen et al. [12] introduced the concept of knowledge distillation to image and text networks, aiming to improve the matching capabilities of two networks with different modalities by balancing information from both networks. Wang et al. [118] addressed the color-dependency issue in cross-modal pedestrian retrieval by proposing a Color-Aware Information Beyond Color(CAIBC) architecture through joint optimization of a multi-branch structure. CAIBC includes three branches(RGB, Grayscale, and Color) employing mutual learning mechanisms for knowledge exchange across branches. It achieved an accuracy of 88.37% in top-10 on the CUHKPEDES dataset for weakly supervised text-to-person retrieval.

## IV. ARCHITECTURE

When mapping features from a specific modality to a common manifold, the feature distribution of other modalities remains imperceptible. This implies that embedding and aligning multimodal features in the common manifold entirely depend on the model's own experience rather than the actual data distribution. In other words, a major challenge in cross-modal Person Re-ID is ensuring that the feature distribution in the common manifold accurately reflects the feature distribution in the original modalities. This necessitates the model to possess sufficient capability to capture and understand the relationships between different modalities, which typically requires abundant data and carefully designed model structures.

## A. Convolutional Neural Network

Text-based person Re-ID can be viewed as a multimodal pedestrian retrieval problem, with the primary challenge lying in the extraction of features from both visual and textual data. Due to the capability of neural networks to automatically learn and extract features from pedestrian images, the cumbersome process of manual feature design can be avoided. Most existing efforts consider employing network architectures designed for image classification as the backbone of visual networks. Consequently, convolutional neural networks (CNNs) have been utilized extensively in text-based pedestrian re-identification, primarily leveraging CNNs [8], [20], [35]-[37], [42], [82], [92], and their major variants such as VGG-16 [17], [69], [75], [94], [101], [116], [124], MobileNet [1], [127], [140], [161], ResNet50 [37], [68], [112], [117], [118], ResNet101 [47], [79], [93], [95], [102], ResNet-152 [18], etc., for extracting crucial information in visual features. These networks are sometimes also employed for text feature extraction [35][37], [92]. The combination of pedestrian images extracted by CNNs and textual description information is achieved, aligning low and high-level semantic information across modalities in an implicit space using matching methods. This facilitates the recognition and matching of pedestrian identities.

Two independently pre-trained ResNet-50 models on ImageNet serve as the visual backbone in study [118]. Addressing the challenge of manual annotation for existing crossmodal data, Jing et al. propose a Matrix Alignment Network (MAN) to tackle cross-modal cross-domain person search tasks, applying the single-modal cross-domain adaptive idea to the problem of image-text-based cross-modal person reidentification [58]. Farooq et al. introduce the Deep Vision Net model based on ResNet-50 to construct the visual feature extraction backbone [37]. Deviating from the original architecture, the proposed network includes two fully connected layers before the classifier layers, applies batch normalization after the two FC layers, and employs dropout before the final layer. They further present a two-stream deep convolutional neural network framework supervised by cross-entropy loss [36]. The language network is modified to resemble the visual branch in terms of the number of layers in deep residual networks, connecting the class probability weights between the second-to-last layer and the final layer, i.e., sharing logits of the softmax layer across both networks. AXM-Net, proposed by [35], is a novel architecture based on convolutional neural networks (CNN) that does not rely on external cues for explicit alignment of feature embeddings and is capable of learning semantically aligned cross-modal feature representations. Most of the aforementioned works initialize pre-trained ResNet50 [48] and ResNet-101 [93] backbones on large datasets to better map modal-corresponding information. Additionally, the application of large models such as CLIP in some fine-grained visual tasks has demonstrated effective paradigm transfer, fully migrating visual-text large models based on contrastive learning to text-based person re-identification tasks [147]. Presently, many studies commonly regard representation learning as a classification method for person re-identification, believing that feature embedding is a core module for text-based person re-identification tasks, crucial for narrowing the semantic gap between modalities. CNNs exhibit significant advantages in perceptual localization of local features in pedestrian vision and end-to-end learning. However, they still have certain limitations in handling long-term dependent data relationships.

## B. Recurrent Neural Network

Beyond visual features, textual descriptions directly impact the accuracy of text-based Person Re-ID. Efficient extraction of textual features is crucial under existing textual data descriptions [29]. Recurrent Neural Networks (RNNs) and their variants, as neural networks for processing sequential events, including text, have been widely accepted in the research community [75], [101], [116], [124]. Early on, in this field, word embedding techniques were initially employed to map words to vector spaces. Subsequently, variants such as LSTM [69], Bi-LSTM [12], [16], [18], [60], [94], [102], [127], [135], [136], [151], [161], [165] attempted to capture long dependencies of sentences for semantic feature extraction from text sequences. Later, with the emergence of large language models like Bidirectional Encoder Representations from Transformers (BERT) [28], the paradigm shift of 'pretraining + fine-tuning' has been triggered in the Text-based Person Re-ID domain. Utilizing pre-trained models to learn general features of natural language and obtaining high-quality word embeddings, followed by inputting into RNNs variants to capture contextual dependencies.

Niu et al. [93] first proposed a concise and effective framework for image-text alignment to address visual-semantic differences. Innovatively integrating two opposing directions, source to target and target to source, for cross-domain adaptation. Wang et al. [117] introduced a novel Adversarial Multi-space Embedding Network (AMEN) to learn and match embeddings in multiple spaces. Bi-GRU is used for text feature extraction, and the paradigms of inter-modal and intra-modal reconstruction collaborate to embed features correctly into the opposite modal space while learning a robust common space. [81] introduces a Text-Graph Attention Network consisting of a Bi-LSTM layer, a text scene graph module, and a joint embedding layer. Bi-LSTM units gradually take the embeddings of each word in the text as input, capturing the historical and future context information of processed words. Yan et al. [151] employed an LSTM-based Recurrent Feature Aggregation Network to accumulate discriminative features from the first LSTM node to the deepest LSTM node, effectively alleviating interference caused by occlusion, background noise, and detection failures. Chen et al. [17] decomposed video sequences into multiple segments and used LSTM to learn to detect the segments where the images are located in both time and space features. This approach reduces variations in the same person across samples, favoring the learning of similarity features. Both of the above methods independently process each video frame. Ding et al. [29] employed Bidirectional Long Short-Term Memory Network (Bi-LSTM) to process each text description, capturing relationships between words. With contextual cues, the Word Attention Module (WAM) based on word representations infers word part correspondences, referring to word part correspondences to obtain original part-level texture features. Wang et al. [129] proposed a new algorithm called LBUL to learn a Consistent Cross-Modal Common Manifold (C3M) for text-based person retrieval. For textual samples, the proposed method embeds the text modality into C3M after considering the distribution features of cross-modalities comprehensively to achieve more reliable cross-modal distribution consensus, rather than blind predictions. However, RNN and its variants also have some issues in the Text-based Person Re-ID task. For instance, the embedding quality of pedestrian text inputs is often very sensitive, features extracted are typically influenced by the length of data sequences, and RNNs only establish temporal correlations on high-level features, thus failing to capture the temporal clues describing the local details of images.

## C. Autoencoder

Sometimes, text-based Person Re-ID tasks also involve the utilization of Autoencoder network architectures. Commonly, these models map images or textual descriptions into a low-dimensional embedding space and employ this embedding space for person re-identification tasks [47], [69]. In this manner, pedestrian images and their corresponding textual descriptions are mapped to a low-dimensional vector space, which encapsulates crucial semantic information, as both image and text provide pivotal details about the identity of the pedestrian. Autoencoders, particularly those with atypical encoder components, exhibit several advantages in Text-based Person Re-ID tasks. Firstly, they can overcome issues prevalent in traditional approaches arising from the lack of semantic information in image-text pairs. Secondly, Autoencoder models proficiently capture essential information from textual descriptions and compress it, facilitating effective data compression. Additionally, this approach boasts good interpretability, as relationships between pedestrian images and textual descriptions can be visualized based on vectors within the embedding space. Wang et al. [117] introduced an innovative Adversarial Multi-space Embedding Network (AMEN) for learning and matching embeddings across multiple spaces. Following an encoder-decoder paradigm, the intermodal reconstruction paradigm collaborates with the intramodal reconstruction paradigm to embed features accurately into the opposite modal space while learning a robust common space. [124] proposed an Improved Embedding Learning through Virtual Attribute Decoupling (iVAD) model. This method enhances embedding learning by using an encoderdecoder structure to decompose attribute information in images and text. Pedestrian attributes are treated as hidden vectors, and attribute-related embeddings are obtained. Diverging from previous works that separate attribute learning from imagetext embedding learning, a hierarchical feature embedding framework is introduced. The Attribute-Enhanced Feature Embedding (AEFE) module merges attribute-related embeddings into the learned image-text embeddings, leveraging attribute information to enhance feature discriminability. However, a drawback of this approach is its reliance on a substantial amount of training data to train the Autoencoder model, and meticulous tuning of model parameters and embedding space dimensions is necessary to achieve optimal performance.

## D. Graph Neural Network

Utilizing Graph Neural Networks (GNN) for text-based pedestrian re-identification has become a noteworthy research

direction in recent years [76], [81]. GNN is a type of deep learning method based on graph structures, suitable for handling unstructured data and relationships, and applicable to various data types such as images, text, and structured data. In text-based Person Re-ID tasks, GNN is often employed to construct a graph structure, where each node represents a pedestrian body part along with its corresponding phrase description. Edges between nodes represent semantic relationships between different body parts and descriptions, such as verb (wear), noun (women), adjective (yellow), and others. The methods utilizing GNN for text-based Person Re-ID typically involve the following steps. First, images and corresponding text descriptions for each pedestrian are extracted from the dataset. Second, two graph convolution structures are established based on auxiliary strategies, such as attention mechanisms, image segmentation, etc. Afterwards, Graph Convolutional Networks (GCN) are employed to learn the relationships between body nodes in the graph structure and transform these relationships into feature vectors. Finally, feature vectors of body structures undergo cross-modal asymmetric alignment, and the text modality is input into a classifier for Person Re-ID.

The methods employing GNN for text-based Person Re-ID offer several advantages. Firstly, they can capture semantic relationships between different pedestrian images and text descriptions, facilitating future inference-based retrieval, i.e., recognizing a pedestrian engaged in complex events. Secondly, GNN can handle text descriptions of varying lengths and structures, providing the advantage of data flexibility. Numerous studies have explored GNN-based Person Re-ID methods, including those using Graph Convolutional Networks (GCN) and methods based on graph self-attention mechanisms. Liu et al. [81] proposed a novel deep adversarial graph attention convolutional network (A-GANet) for text-based person search. A-GANet utilizes text and visual scene graphs, including object properties and relationships, from pedestrian text queries and gallery images to learn information-rich text and visual representations. It learns an effective joint text-visual latent feature space through adversarial learning, bridging the modality gap and promoting person matching. Li et al. [76] presented a fundamental framework using graph convolution as a feature representation. They simulate adversarial attacks caused by text and image diversity on feature extraction by embedding additional attack nodes in graph convolution layers to enhance the model's robustness to textual and visual diversity. However, when graph neural networks compute edges for pedestrian body nodes and entity relationships, it may result in high computational complexity and memory requirements. Moreover, GNNs are more susceptible to overfitting compared to general network structures.

## E. Transformer

Due to the success of Transformers in multimodal tasks, it has become a popular architecture widely employed in text-based Person Re-ID tasks [20], [40], [57], [70], [79], [106], [121], [147], [150]. Currently, a commonly used approach involves a dual-stream network architecture, where one stream is dedicated to processing pedestrian images, and the other handles the corresponding text descriptions. In the image stream, convolutional neural networks [173] or vision transformers [107] are utilized to extract visual features from pedestrian images. Simultaneously, in the text stream, Long Short-Term Memory (LSTM) [140] or transformers [40] are applied to extract feature vectors from the corresponding text descriptions. Li et al. [79] introduced a transformer-based Person Re-ID architecture, evaluating the shared attention between language reference terms and visual features through transformer blocks. Apart from achieving outstanding search performance, the proposed method also provides interpretability by visualizing the attention between image parts in person pictures and their corresponding referential terms. Existing works often overlook the granularity differences between features of the two modalities, where visual features tend to be fine-grained while text features are coarse-grained, leading to significant modality gaps. Shao et al. [106] proposed an end-to-end framework named Learning Granularity-Unified Representations (LGUR), based on the transformer, aiming to map visual and text features to a unified granularity space. Addressing alignment deficiencies between image region features and text features in weakly supervised learning methods, resulting in poor feature distribution, Gao et al. [40] presented a novel method called Conditional Feature Learning based Transformer (CFLT). CFLT maps sub-regions and phrases to a unified latent space and dynamically adjusts features from one modality based on data from another modality by constructing explicit conditional embeddings.

With the significant progress achieved by large models, Sarafianos et al. [102] demonstrated for the first time the effectiveness of BERT as an openly accessible word embedding extraction language model in the domain of image-text matching. Subsequently, they combined two feature vectors using a cross-attention mechanism and applied them to pedestrian re-identification tasks. Li et al. [70] proposed a semantically aligned feature aggregation network that adaptively aggregates unit features with the same semantics into different partperceptive features. Initially introducing Transformer-based backbones ViT [31] and BERT into visual and text modalities, Ji et al. [76] extracted robust feature representations for textbased person search. They proposed an Asymmetric CrossScale Alignment (ACSA) method, employing Swin Transformer for visual representation extraction and BERT for text and phrase representation in the text domain. In contrast to previous methods that use two separate networks for feature extraction, Wang et al. [118] introduced an Implicit VisualText (IVT) framework, a single-stream network architecture using only a single transformer. Text and image feature extraction utilizes a backbone network, optimizing network shared parameters with cross-modal data to facilitate learning a common space mapping. Recently, Vision-Language Pretraining (VLP) models in visual language pre-training have demonstrated the transferability of knowledge for downstream Text-Picture Search (TPS) tasks, enhancing performance effectively. An approach called CLIP-driven Fine-grained Information Mining (CFine) [121] was proposed to leverage CLIP's visual knowledge for person re-identification, incorporating fine-grained information mining to explore intra-modal dis-

criminative clues and inter-modal correspondences. However, the lack of VLP text representations may compromise the valuable cross-modal alignment of the two encoders. Hence, Wang et al. [121] introduced a model with two pre-training modes, TP-TPS, where both text and image encoders are based on transformers, and all backbones are initialized with pretraining weights from CLIP. They introduced Multi-Integrity Description Constraints (MIDC) to enhance the robustness of the text modality by combining different components of a finegrained corpus during training. The advantage of the Transformer model lies in its ability to effectively learn sequence information, making it capable of capturing information in text descriptions efficiently. Additionally, this approach exhibits good interpretability in the field of multi-scale learning, as it allows the visualization of regions of interest in pedestrian images and keywords in text descriptions based on attention weights. Finally, it plays a crucial role in unified modality architectures.

## V. OPTIMIZATION

Deep metric learning plays a paramount optimization role in cross-modal person re-identification [166]. For instance, it enhances the discriminative power of feature representation, addresses a multitude of noise and variations such as diverse lighting conditions, poses, and occlusions, adapts the distance metric accordingly, embeds more semantically meaningful spatial structures, and leverages domain adaptation to tackle the challenges posed by multiple viewing angles. Of particular significance is the optimization of loss functions, thereby improving the matching performance in cross-modal person reidentification tasks [32]. This approach effectively addresses a spectrum of challenges, elevating the network's robustness and generalization capacity, consequently achieving more accurate person matching across distinct camera scenes and conditions. This study places emphasis on prevalent loss functions used in text-based person re-identification tasks, including identity loss, verification loss, contrastive loss, triplet loss, quadruplet loss, and adversarial loss, as shown in Figure 3.

## A. Identity Loss

Zheng et al. [170] introduced the ID Discriminative Embedding (IDE) network, which treats each pedestrian as a separate category and employs the pedestrian's ID as a classification label to train a deep neural network. This allows the network to learn the capability of predicting the identity given a pedestrian image. The classification loss is typically computed using the cross-entropy loss function. During training, the classifier optimizes its weight parameters by minimizing the classification loss. A fully connected layer (FC) for classification is appended to the end of the network, and the softmax activation function maps the features to a probability space representing identity labels [42]. The cross-entropy loss for the multi-class pedestrian Re-ID task can be expressed as:

$$L _ { I D } = - \frac { 1 } { N } \sum _ { i = 1 } ^ { N } y _ { i } \log ( p _ { i } ). \quad \quad ( 4 ) \quad \text{$\text{$level$}$}.$$

Here, x i denotes the input image, y i represents the corresponding label, p i is the probability that x i is recognized as y i after the softmax function, and N indicates the batch size. Identity loss is widely adopted in pedestrian re-identification [31], [36], [37], [87], [102], [112], [117], [118], [133], due to its automatic hard sample mining and ease of training. However, as the labeled data increases significantly, issues related to decreased efficiency in classifier training might arise, especially when dealing with datasets containing a large number of similar pedestrians. In cross-modal pedestrian reidentification, the classification loss is often used in conjunction with other loss functions [47], [165], such as contrastive loss and triplet loss [69], [118], [151], [168], to enhance performance of the model.

## B. Verification Loss

The verification loss is employed to quantify the similarity between two pedestrian images [171]. In pedestrian reidentification, it is often necessary to compare two images to determine if they belong to the same individual. The verification loss aims to achieve this objective by comparing the feature vectors of two images to compute their similarity [166]. Specifically, the verification loss is calculated by inputting the feature vectors of two images into a similarity metric function. This similarity metric function can be Euclidean distance, cosine similarity, or other similar measures. If two images belong to the same person, their feature vectors should exhibit high similarity, resulting in a high similarity score [180]. Thus, the verification loss is minimized to enhance the similarity between two images of the same person. The specific formula is as follows:

$$\underset { \dots } { \text{ate} } \underset { \dots } { \text{ns.} } \quad L _ { v e r i } = - \frac { 1 } { N } \sum _ { i = 1 } ^ { N } y _ { i } \log ( p _ { i } ) + ( 1 - y _ { i } ) \log ( 1 - p _ { i } ). \quad ( 5 )$$

Here, N is the number of sample pairs, y i is a label indicating whether the sample pair x i and x j belongs to the same category ( 1 denotes the same category, 0 denotes different categories), and the softmax function is utilized to calculate the probability p i of having the same label for both samples. Due to the necessity of pairwise computation, the verification loss tends to have lower efficiency in re-identification tasks. Therefore, it is often used in combination with other losses, such as identity loss, to leverage the advantages of both methods, thereby enhancing recognition speed and accuracy.

## C. Contrastive Loss

Contrastive Loss was originally introduced by Sumit Chopra et al. in the context of face verification tasks [24], aiming to train neural networks to determine whether two inputs belong to the same facial identity. In the context of person re-identification, the principle of encouraging similar identity feature representations to be closer in the feature space while dispersing feature representations of different identities is leveraged for model training within Siamese networks [131]. The Contrastive Loss not only considers the distances between

Fig. 3: The figure shows the principles of the most prominent alignment loss methods in text-based person re-identification. These include identity loss, verification loss, comparison loss, triplet loss, quadruplet loss and adversarial loss. The vast majority of new alignment methods being devised are currently based on variants or combinations of these losses.

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00096v1-figure-bf0f89f6-7e15-4154-a2ec-22a1a64a4add.jpg)

images of the same identity but also takes into account the distances between images of different identities. The function can be expressed as follows:

apart. By adjusting the value of margin and the weight of the loss, the distance range between sample pairs can be controlled, influencing the learning of feature representations [115].

$$L _ { c o n } = \frac { 1 } { 2 N } \sum _ { n = 1 } ^ { N } y _ { n } D _ { n } ^ { 2 } + ( 1 - y _ { n } ) \max ( m a r g i n - D _ { n }, 0 ) ^ { 2 }. \quad D _ { \ } T r i _ { i }$$

(6)

Here, N is the number of sample pairs, y n is a label indicating whether sample pairs x i and x j belong to the same class (1 for the same class, 0 for different classes), D n represents the distance between feature representations (Euclidean distance, cosine distance, etc.), and margin is a threshold indicating that when sample pairs belong to different classes, their distance should be greater than this value. For sample pairs belonging to the same class, the loss is the square of the distance D n , while for sample pairs belonging to different classes, the loss is the square of the difference between the distance and the threshold value margin . This framework aims to bring feature representations of sample pairs from the same class closer together and push feature representations of sample pairs from different classes further

## D. Triplet Loss

The Triplet Loss is a widely utilized loss function in pedestrian re-identification, aiming to learn discriminative feature representations by comparing the distances between three images [11], [35], [49], [52], [59], [117], [118], [118], [133], [135], [151]. Its fundamental concept involves learning feature representations with discrimination, such that the distance between images of the same person is minimized, while the distance between images of different individuals is maximized [8], [42], achieved by minimizing the Triplet Loss during training. Specifically, given an anchor image, a positive sample image, and a negative sample image, the objective of Triplet Loss is to minimize the distance between the anchor and the positive sample while simultaneously maximizing the distance between the anchor and the negative sample [168], [127]. The formal definition of Triplet Loss is as follows:

$$L _ { t r i p l e t } = & \sum _ { n = 1 } ^ { N } \max ( | | f ( x _ { n } ^ { a } ) - f ( x _ { n } ^ { p } ) | | _ { 2 } ^ { 2 } & \stackrel { \searrow } { \text{and set} } } \\ & - | | f ( x _ { n } ^ { a } ) - f ( x _ { n } ^ { n } ) | | _ { 2 } ^ { 2 } + m a r g i n, 0 ). & \stackrel { \searrow } { \text{represents} } } \\ - - \quad. & \quad. & \quad. & \quad. & \quad. & \quad \text{in dist} ^ { 2 }$$

Here, f is a function that maps an image x to a feature vector, x a n , x p n and x n n are the anchor, positive, and negative sample images of the n -th instance, respectively. || · || 2 represents the Euclidean distance between two embedding vectors, max( ) · represents taking the maximum value between 0 and · , margin is a hyperparameter controlling the difference in distances between images of the same person and images of different people. It is worth noting that in practice, Triplet Loss has encountered certain issues, such as sample selection and slow learning speed. Consequently, improved versions have been proposed [35], [69], [118]. Wang et al. [127] proposed a novel mutually connected classification loss (MCCL) based on triplet loss to fully exploit identity-level information. It not only incorporates identification information into both images and language descriptions but also encourages similar crossmodal classification probabilities for instances belonging to the same identity. These improved versions prove to address certain existing issues more effectively and have demonstrated excellent performance in person re-identification tasks.

## E. Quadruplet Loss

The Quadruplet Loss is an extension of the Triplet Loss [21]. The Triplet Loss is commonly employed to train a neural network to embed images of the same person into close proximity within the embedding space, while images of different individuals are embedded far from the space occupied by the same person. However, the Triplet Loss may suffer from issues such as gradient vanishing or exploding, which could render it less stable in practical usage. The Quadruplet Loss enhances the stability of the Triplet Loss by introducing a fourth sample.

In specific terms, the Quadruplet Loss computes pairs of embedding distances for two images of the same person and two images of different individuals, and employs these four distances to formulate the loss function. In this regard, the two images of the same person should be closer, while the two images of different individuals should be farther apart. By separately using the two images of the same person and the two images of different individuals to calculate distances, the Quadruplet Loss can yield more stable training, thus enhancing the performance of person re-identification. The formula for the Quadruplet Loss is as follows:

$$L _ { q u a d } = \sum _ { n = 1 } ^ { N } \max ( & | | f ( x _ { n } ^ { a } ) - f ( x _ { n } ^ { p } ) | | _ { 2 } ^ { 2 } - | | f ( x _ { n } ^ { a } ) - f ( x _ { n } ^ { n 1 } ) | | _ { 2 } ^ { 2 } & \text{discrimi} \\ & + \alpha, 0 ) + \max ( | | f ( x _ { n } ^ { a } ) - f ( x _ { n } ^ { n 1 } ) | | _ { 2 } ^ { 2 } & \text{module} \\ & - | | f ( x _ { n } ^ { n 1 } ) - f ( x _ { n } ^ { n 2 } ) | | _ { 2 } ^ { 2 } + \beta, 0 ). & \text{from im} \\ & \text{Here, } f \text{ denotes the embedding function of the neural} & \text{ reliably}$$

Here, f denotes the embedding function of the neural network, x a n , x p n , x n 1 n and x n 2 n respectively represent the anchor image, positive sample image, first negative sample image, and second negative sample image for the n -th sample. || · || 2 represents the Euclidean distance between two embedding vectors. max( ) · represents taking the maximum value between 0 and · , The hyperparameters α and β control the difference in distances between the two images of the same person and the two images of different individuals.

The Quadruplet Loss effectively reduces the within-class variance and improves the discriminative performance of the model by introducing an additional negative sample. However, the increase in parameters also requires the computation of more pairs of samples, leading to a significant increase in computational complexity and longer training time. Therefore, suitable strategies are required to select effective quadruples in practical applications.

## F. Adversarial Loss

The adversarial loss primarily leverages adversarial training to learn more robust pedestrian feature representations [76]. Its fundamental concept involves mapping pedestrian feature embeddings from the training data into a similarity space, where the distribution of generated image features is made closer to that of real image features, enhancing the robustness of the network in handling multi-modal information [102]. Specifically, adversarial loss employs adversarial samples during training, where the model seeks to minimize the adversarial loss. The computation of the adversarial loss involves introducing perturbations to the model input, making the input pedestrian image appear visually similar to the original image while introducing significant disparities in the embedding space. These perturbations are generated by a generator network, which attempts to produce images similar in appearance to the input image but with substantial deviations in the embedding space. The discriminator seeks to maximize the adversarial loss to effectively discriminate between these two types of features. The general formula for adversarial loss is as follows:

$$L _ { a d v } = & \mathbb { E } _ { x \sim p _ { d a t a } ( x ) } [ \log D ( x ) ] \\ & + \mathbb { E } _ { z \sim p _ { z } ( z ) } [ \log ( 1 - D ( G ( z ) ) ) ].$$

Here, G is the generator, D is the discriminator, x represents real image features, z represents generated image features, log D x ( ) denotes the logarithm of the discriminator's predicted probability for real image features, and log(1 -D G z ( ( ))) denotes the logarithm of the discriminator's predicted probability for generated image features. Liu et al. [81] proposed the application of adversarial principles in pedestrian re-identification to acquire more robust feature representations. They contend that the distributions of text and image features should be modality-invariant and semantically discriminative, while maintaining latent cross-modal semantic similarities. These features exhibit enhanced resistance to adversarial attacks. Subsequently, an adversarial learning module was devised to optimize the representations extracted from image graph attention networks and text graph attention networks. The objective of the modality discriminator is to reliably distinguish between modalities of samples as much as possible, given a well-trained feature transformer, until the

point where discrimination becomes infeasible. Adversarial loss is employed to classify the modality labels (belonging to either image or text) of input samples. Despite its relatively recent application in pedestrian re-identification, adversarial loss has demonstrated significant efficacy in some recent studies [117].

## VI. CHALLENGE

In this paper, we first give an overview of text-based person Re-ID in two parts: attribute-based and natural language-based person Re-ID. Secondly, the datasets based on the two parts are summarized and described respectively. Moreover, the commonly used technical strategies in text-based person ReID are analyzed. In addition, an overview of the deep learning architectures that usually appear in text-based person Re-ID is made. Further, the loss functions (alignment optimization methods) commonly used in text-based person Re-ID are summarized and described in detail. Finally, we analyze the challenges and possible future directions of text-based person ReID from the perspectives of feature extraction, data scarcity, pre-trained models, and generative models, respectively.

Although text-based person Re-ID technology offers numerous advantages in fields such as video surveillance, security, social media, and virtual reality [41], it still faces various challenges. The most critical issues include the ambiguity and granularity of natural language. Identical descriptions may be incomplete, ambiguous, or contain ambiguities, potentially leading to incorrect identity matches. On the other hand, photographs available in real-world galleries pertaining to pedestrians often lack high resolution, ample lighting, and freedom from obstruction [36]. Zhang et al. [192] demonstrated recently that gait information may be the basis for general person Re-ID, and that its features may be more robust compared to the highly homogeneous pedestrian appearance feature signature. However, existing gait recognition methods are not able to meet the requirements of less controlled scenarios, e.g., most of the studies are based on well controlled scenarios such as narrow corridors. If one wants to improve the efficiency of person Re-ID by combining gait features with appearance features, it may be necessary to obtain more coherent and high-quality gait data, while the appearance is recognizable. Therefore, integrating text descriptions with image/video information requires more efficient and robust cross-modal fusion techniques for the system to better understand and match entities. Pre-trained large-scale multimodal models may represent a reasonable direction for achieving improved encoding effects. Furthermore, text-based person reidentification necessitates more high-quality data to enable the model to learn relationships between various descriptions and identities [150]. It is noteworthy that the recent popularity of text-to-image generation models, as a novel approach in pedestrian re-identification, holds significant potential, as these models generate graphics from textual descriptions for retrieval. Given that cameras in the real world do not always capture images from an available query library, customizing the generation of corresponding target entities based on eyewitness text is meaningful. However, due to the insufficient maturity of text-to-image generation model controllability, we remain optimistic about this direction.

## VII. FORESIGHT

## A. A Novel Baseline

Since the current text-based person Re-ID tasks being based on disparate datasets, specialized models are separately trained on large samples. However, in the real world, the lack of a uniform style (resolution, size, angle, brightness) in pedestrian Re-ID data is often due to the constraints imposed by different specifications of data collection devices (cameras). Retrieving arbitrary pedestrian targets in an open-world scenario becomes quite challenging for dedicated models. There is an urgent need for a new approach that enables zero-shot retrieval of targets from any gallery. On the other hand, text-to-image retrieval fundamentally relies on comparing cross-modal data distributions from small datasets through contrastive learning. If the gallery itself lacks target pedestrians, this results in ineffective retrieval, contradicting the concept of open-world pedestrian recognition. Inspired by the controllable generation retrieval in text-to-image for pedestrians, we propose a novel baseline model for text-based person re-identification, which involves re-retrieving target images from text to image.

As illustrated in Figure 4, the entire text-based pedestrian image generation guided Re-ID baseline architecture is demonstrated. The initial step involves encoding natural language descriptions of pedestrians and utilizing a diffusion model to generate corresponding pedestrian images. Subsequently, these generated pedestrian images are employed in the task of pedestrian re-identification. Given that current text-to-image generation models primarily focus on coarsegrained overall image generation tasks, emphasizing inter-class coordination among different objects rather than concentrating on intra-class distinctiveness in human images, and while simultaneously considering authenticity, a strategy based on human body key-point assistance is adopted. This strategy involves extracting features from human body key-points to assist in controlling the pedestrian generation process, ensuring compliance with the requirement for the presence of the majority of the human body in the pedestrian re-identification task. The generated pedestrian images serve as queries for retrieval and are input into the backbone network for visual feature extraction. To enhance retrieval accuracy, emphasis is placed on restoring features with the highest relevance to the original image. For instance, in comparison to generated images, sensitivity to channel attention may be more pronounced in original images. Various attention mechanisms are employed to assess their impact on retrieval effectiveness. However, due to the coarser granularity of linguistic descriptions compared to images, the generation of pedestrian images may not be sufficiently detailed in terms of the style and material of the pedestrian's clothing. For example, the textual description may be as broad as 'black coat', but the image may be as specific as 'short black leather jacket', which results in feature learning inaccuracy. In order to improve the accuracy of localized feature generation for pedestrians in a realistic open set, an approach using cross-attention based image editing is

TABLE II: The table demonstrates the results of person Re-ID with and without fine-tuning the stable diffusion model. Tests are performed on three datasets, CUHK-PEDES, RSTPReID and ICFG-PEDES, respectively, using different visual representation backbone networks.

| Stable Diffusion   | Method                | CUHK-PEDES           | CUHK-PEDES   | CUHK-PEDES           | RSTPReID             | RSTPReID             | RSTPReID             | ICFG-PEDES           | ICFG-PEDES    | ICFG-PEDES           |
|--------------------|-----------------------|----------------------|--------------|----------------------|----------------------|----------------------|----------------------|----------------------|---------------|----------------------|
| Stable Diffusion   |                       | rank@1               | rank@5       | rank@10              | rank@1               | rank@5               | rank@10              | rank@1               | rank@5        | rank@10              |
|                    | DenseNet EfficientNet | 0.0396 0.0584 0.0630 | 0.1072       | 0.1944 0.2216 0.2460 | 0.0275 0.0392 0.0471 | 0.0980 0.1255 0.1294 | 0.1882 0.2000 0.2235 | 0.0077 0.0087 0.0084 | 0.0210 0.0209 | 0.0323 0.0326 0.0326 |
|                    |                       |                      | 0.1233       |                      |                      |                      |                      |                      |               |                      |
|                    | Swin Transformer      |                      | 0.1519       |                      |                      |                      |                      |                      | 0.0215        |                      |
|                    | ResNet                | 0.0674               | 0.1794       |                      | 0.0491               | 0.1353               |                      |                      | 0.0287        | 0.0392               |
|                    |                       |                      |              | 0.2578               |                      |                      | 0.2348               | 0.0103               |               |                      |
|                    | DenseNet              | 0.1101               | 0.2179       | 0.3298               | 0.0745               | 0.2196               | 0.3176               | 0.0328               | 0.0783        | 0.1278               |
|                    | EfficientNet          | 0.1498               | 0.2619       | 0.4305               | 0.1103               | 0.2784               | 0.3686               | 0.0651               | 0.1254        | 0.1701               |
|                    | Swin Transformer      | 0.1622               | 0.3794       | 0.4675               | 0.1098               | 0.2667               | 0.3804               | 0.0632               | 0.1302        | 0.2015               |
|                    | ResNet                | 0.1586               | 0.3648       | 0.4827               | 0.1176               | 0.2824               | 0.4118               | 0.0749               | 0.1306        | 0.2191               |

recommended. After using natural language instructions, the local features of pedestrians are controlled to change, and the witness's memory of the local features for the target person is refined and specialized. Finally, the results of pedestrian retrieval are obtained through different optimization strategies.

extracted through skip connections and zero convolutions, facilitating their incorporation into pedestrian generation.

Fig. 4: Text-Based Pedestrian Image Generation Guided Re-ID baseline architecture. The entire architecture consists of three modules: A) Text-based pedestrian generator, B) Pedestrian refinement module, and C) Pedestrian re-recognition system. The entire architectural flow can be viewed starting with module A at the bottom left, preferably in a clockwise direction. Note that there may not be some kind of very clear boundary between modules A and B, and that there is interaction between them in the pedestrian generation process.

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00096v1-figure-83fd16b6-cef1-4852-9293-50874f46f55c.jpg)

## B. Experiment &amp; Results

For A) Text-based pedestrian generation module , We employed the Frozen CLIP Text Encoder [100] to encode pedestrian descriptions, followed by inputting them into stable diffusion [187] to generate pedestrian images. The denoising module utilized the cross-attention U-Net [188] backbone, which proves beneficial for the generation of high-quality pedestrian images. For pedestrian pose assistance, we adopt ControlNet [189] for Pedestrian integrity control, which ensures that the generated pedestrian image has full body. Specifically, OpenPose [184] was employed for detecting key points on the pedestrian body. Subsequently, pose features were

In the part of B) pedestrian local feature refinement , we input the local image editing instructions and then use GPT-3 [190] to rewrite and replace the pedestrian image generation instructions. Re-inputting the rewritten text cue and the pedestrian image before rewriting into the stable diffusion model, the visual and textual features of pedestrians are fused using the cross-attention layer, and the image can be edited by editing the text only to realize the mapping from the source image to the target image [191]. As a comparative analysis, we also attempted fine-tuning the baseline model using a subset of text-image pairs data based on text-driven pedestrian reidentification. In the retrieval module, the generated pedestrian images were employed as queries.

In the C) pedestrian Re-ID system , ResNet-50 [48] served as the visual backbone, incorporating self-attention mechanisms. The Part-based Convolutional Baseline (PCB) [110] strategy was employed for segmenting human bodies to extract feature maps, and random horizontal image flipping was applied for data augmentation. The training process involved the use of Instance Batch Normalization (IBN) [185]strategy, with a batch size set to 32, droprate at 0.5, erasing probability at 0.2, ins-gamma set to 32, learning rate at 0.05, and weight decay at 0.0005. Furthermore, the training amalgamated ID loss, Contrastive loss, and Triplet loss as the loss functions. The optimization utilized SGD optimizera with momentum, and the model was trained for 60 epochs.

As depicted in the Table II, we evaluated the proposed TextBased Pedestrian Image Generation Guided Re-ID (TBPGR) baseline architecture on three widely used text-based person Re-ID datasets (CUHK-PEDES, RSTPReid, ICFG-PEDES). Specifically, retrieval of pedestrian images was generated using the generation modules with and without fine-tuned Stable Diffusion, respectively. We report the performance of visual representation modules DenseNet, EfficientNet SwinTransformer and ResNet on the task for each dataset and each condition of with and without fine tuning. ResNet generally shows better performance. After the image generation in the intermediate step, we qualitatively analyzed the content of the images. It was found that the quality of the pedestrian images generated before fine-tuning was higher, and the quality of the images generated after fine-tuning became lower, but the retrieval efficiency was slightly improved, conjecturing that the quality of the pedestrian image text data pairs had some

Fig. 5: Generated images of text-based pedestrians without and with fine-tuning are shown, along with their retrieval Rank@10 results. The green numbers shown at the top of the image represent a positive retrieval and the red color represents a negative retrieval.

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00096v1-figure-7e450731-c6e6-4a79-83ad-2997b708d0ce.jpg)

effect on the data distribution. Besides, since the CUHKPEDES dataset retrieval effect is better than other datasets, it is hypothesized that it is because the generation of pedestrian images seldom involves the factors such as lens occlusion and illumination changes, which have some implications on the pedestrian retrieval performance, as shown in Figure 5.

Although our baseline model(TBPGR) is still not powerful enough in comparison to traditional training approaches based on implicit spatial alignment of text and images, we believe that such an open-retrieval paradigm is far more valuable in future Open Set text-based person Re-ID scenarios than just pursuing the accuracy of the present moment.

## ACKNOWLEDGMENTS

The authors would like to thank all the anonymous reviewers and associate editors for their valuable comments to improve the paper. The authors would also like to thank the authors of the three publicly available datasets.

## REFERENCES

- [1] S. Aggarwal, R. V. Babu, and A. Chakraborty, 'Text-based Person Search via Attribute-aided Matching,' in 2020 IEEE Winter Conference on Applications of Computer Vision (WACV) , pp. 2606-2614, Mar. 2020.
- [2] 'ALIGN: Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision -Google AI Blog.' https://ai.googleblog.com/2021/05/align-scaling-up-visual-andvision.html.
- [3] Y. Bai, M. Cao, D. Gao, Z. Cao, C. Chen, Z. Fan, L. Nie, and M. Zhang, 'RaSa: Relation and Sensitivity Aware Representation Learning for Textbased Person Search,' May 2023.
- [4] A. Bedagkar-Gala and S. K. Shah, 'A survey of approaches and trends in person re-identification,' Image and Vision Computing , vol. 32, pp. 270286, Apr. 2014.
- [5] S. Bird, 'NLTK: The natural language toolkit,' in Proceedings of the COLING/ACL on Interactive Presentation Sessions , COLING-ACL '06, (USA), pp. 69-72, Association for Computational Linguistics, July 2006.
- [6] H.-Q. Cai, X. Li, Y. Ji, Y. Li, and C.-P. Liu, 'Parallel Data Augmentation for Text-based Person Re-identification,' in 2022 International Joint Conference on Neural Networks (IJCNN) , pp. 1-8, July 2022.
- [7] Y.-T. Cao, J. Wang, and D. Tao, 'Symbiotic Adversarial Learning for Attribute-Based Person Search,' in Computer Vision - ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XIV , (Berlin, Heidelberg), pp. 230-247, Springer-Verlag, Aug. 2020.
- [8] 'Cascade Attention Network for Person Search: Both Image and Text-Image Similarity Selection.' https://deepai.org/publication/cascadeattention-network-for-person-search-both-image-and-text-imagesimilarity-selection, Sept. 2018.
- [9] H. Chahar and N. Nain, 'A Study on Deep Convolutional Neural Network Based Approaches for Person Re-identification,' in Pattern Recognition and Machine Intelligence (B. U. Shankar, K. Ghosh, D. P. Mandal, S. S. Ray, D. Zhang, and S. K. Pal, eds.), Lecture Notes in Computer Science, (Cham), pp. 543-548, Springer International Publishing, 2017.
- [10] S. Changpinyo, P. Sharma, N. Ding, and R. Soricut, 'Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize LongTail Visual Concepts,' in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pp. 3558-3568, 2021.
- [11] G. Chechik, V. Sharma, U. Shalit, and S. Bengio, 'Large Scale Online Learning of Image Similarity Through Ranking,' The Journal of Machine Learning Research , vol. 11, pp. 1109-1135, Mar. 2010.
- [12] Y. Chen, R. Huang, H. Chang, C. Tan, T. Xue, and B. Ma, 'CrossModal Knowledge Adaptation for Language-Based Person Search,' IEEE Transactions on Image Processing , vol. 30, pp. 4057-4069, 2021.
- [13] D. Chen, M. Wang, H. Chen, L. Wu, J. Qin, and W. Peng, 'Cross-Modal Retrieval with Heterogeneous Graph Embedding,' in Proceedings of the 30th ACM International Conference on Multimedia , MM'22, (New York, NY, USA), pp. 3291-3300, Association for Computing Machinery, Oct. 2022.
- [14] C. Chen, P. Li, M. Sun, and Y. Liu, 'End-to-End Unsupervised Visionand-Language Pre-training with Referring Expression Matching,' in Conference on Empirical Methods in Natural Language Processing , 2022.
- [15] D. Cheng, Y. Gong, S. Zhou, J. Wang, and N. Zheng, 'Person Reidentification by Multi-Channel Parts-Based CNN with Improved Triplet Loss Function,' in 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , pp. 1335-1344, June 2016.

- [16] D. Chen, H. Li, X. Liu, Y. Shen, J. Shao, Z. Yuan, and X. Wang, 'Improving Deep Visual Representation for Person Re-identification by Global and Local Image-language Association,' in Proceedings of the European Conference on Computer Vision (ECCV) , pp. 54-70, 2018.
- [17] T. Chen, C. Xu, and J. Luo, 'Improving Text-Based Person Search by Spatial Matching and Adaptive Threshold,' in 2018 IEEE Winter Conference on Applications of Computer Vision (WACV) , pp. 1879-1887, Mar. 2018.
- [18] W. Chen, Y. Liu, E. M. Bakker, and M. S. Lew, 'Integrating information theory and adversarial learning for cross-modal retrieval,' Pattern Recognition , vol. 117, p. 107983, Sept. 2021.
- [19] D. Chen, A. Doering, S. Zhang, J. Yang, J. Gall, and B. Schiele, 'Keypoint Message Passing for Video-Based Person Re-identification,' Proceedings of the AAAI Conference on Artificial Intelligence , vol. 36, pp. 239-247, June 2022.
- [20] Y. Chen, G. Zhang, Y. Lu, Z. Wang, and Y. Zheng, 'TIPCB: A simple but effective part-based convolutional baseline for text-based person search,' Neurocomputing , vol. 494, pp. 171-181, July 2022.
- [21] W. Chen, X. Chen, J. Zhang, and K. Huang, 'Beyond triplet loss: A deep quadruplet network for person re-identification,' Apr. 2017.
- [22] Y.-C. Chen, L. Li, L. Yu, A. El Kholy, F. Ahmed, Z. Gan, Y. Cheng, and J. Liu, 'UNITER: UNiversal Image-TExt Representation Learning,' in Computer Vision - ECCV 2020 (A. Vedaldi, H. Bischof, T. Brox, and J.-M. Frahm, eds.), Lecture Notes in Computer Science, (Cham), pp. 104120, Springer International Publishing, 2020.
- [23] Z. Chenyang, F. Jun, and W. Jiaqing, 'Text-to-Image Person Search Based on SSAN Model and Re-rank Post-Processing,' in 2022 International Conference on Computer Engineering and Artificial Intelligence (ICCEAI) , pp. 409-413, July 2022.
- [24] S. Chopra, R. Hadsell, and Y. Lecun, Learning a Similarity Metric Discriminatively, with Application to Face Verification , vol. 1. July 2005.
- [25] M. Cormier, A. Specker, J. C. S. Jacques, L. Florin, J. Metzler, T. B. Moeslund, K. Nasrollahi, S. Escalera, and J. Beyerer, 'UPAR Challenge: Pedestrian Attribute Recognition and Attribute-based Person Retrieval Dataset, Design, and Results,' 2023 IEEE/CVF Winter Conference on Applications of Computer Vision Workshops (WACVW) , pp. 166-175, Jan. 2023.
- [26] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, 'ImageNet: A large-scale hierarchical image database,' in 2009 IEEE Conference on Computer Vision and Pattern Recognition , pp. 248-255, June 2009.
- [27] Y. DENG, P. Luo, C. C. Loy, and X. Tang, 'Pedestrian Attribute Recognition At Far Distance,' in Proceedings of the 22nd ACM International Conference on Multimedia , MM '14, (New York, NY, USA), pp. 789792, Association for Computing Machinery, Nov. 2014.
- [28] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, 'BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding,' May 2019.
- [29] Z. Ding, C. Ding, Z. Shao, and D. Tao, 'Semantically Self-Aligned Network for Text-to-Image Part-aware Person Re-identification,' ArXiv , July 2021.
- [30] Q. Dong, X. Zhu, and S. Gong, 'Person Search by Text Attribute Query As Zero-Shot Learning,' 2019 IEEE/CVF International Conference on Computer Vision (ICCV) , pp. 3651-3660, Oct. 2019.
- [31] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, and N. Houlsby, 'An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale,' in International Conference on Learning Representations , Mar. 2023.
- [32] Y. Duan, J. Lu, J. Feng, and J. Zhou, 'Deep Localized Metric Learning,' IEEE Transactions on Circuits and Systems for Video Technology , vol. 28, pp. 2644-2656, Oct. 2018.
- [33] M. Engilberge, L. Chevallier, P. Perez, and M. Cord, 'Finding Beans in Burgers: Deep Semantic-Visual Embedding with Localization,' in 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition , (Salt Lake City, UT), pp. 3984-3993, IEEE, June 2018.
- [34] F. Faghri, D. J. Fleet, J. Kiros, and S. Fidler, 'VSE++: Improving VisualSemantic Embeddings with Hard Negatives,' in British Machine Vision Conference , July 2017.
- [35] A. Farooq, M. Awais, J. Kittler, and S. S. Khalid, 'AXM-Net: Implicit Cross-Modal Feature Alignment for Person Re-identification,' in Proceedings of the AAAI Conference on Artificial Intelligence , vol. 36, pp. 4477-4485, June 2022.
- [36] A. Farooq, M. Awais, F. Yan, J. Kittler, A. Akbari, and S. S. Khalid, 'A Convolutional Baseline for Person Re-Identification Using Vision and Language Descriptions,' Feb. 2020.
- [37] A. Farooq, M. Awais, J. Kittler, A. Akbari, and S. S. Khalid, 'Cross Modal Person Re-identification with Visual-Textual Queries,' in 2020
- IEEE International Joint Conference on Biometrics (IJCB) , pp. 1-8, Sept. 2020.
- [38] S. Frolov, T. Hinz, F. Raue, J. Hees, and A. Dengel, 'Adversarial Textto-Image Synthesis: A Review,' Neural Networks , vol. 144, pp. 187-209, Dec. 2021.
- [39] H. Galiyawala and M. S. Raval, 'Person retrieval in surveillance using textual query: A review,' Multimedia Tools and Applications , vol. 80, pp. 27343-27383, July 2021.
- [40] C. Gao, G. Cai, X. Jiang, F. Zheng, J. Zhang, Y. Gong, F. Lin, X. Sun, and X. Bai, 'Conditional Feature Learning Based Transformer for TextBased Person Search,' IEEE Transactions on Image Processing , vol. 31, pp. 6097-6108, 2022.
- [41] C. Gao, G. Cai, X. Jiang, F. Zheng, J. Zhang, Y. Gong, P. Peng, X.W. Guo, and X. Sun, 'Contextual Non-Local Alignment over Full-Scale Representation for Text-Based Person Search,' ArXiv , Jan. 2021.
- [42] J. Ge, G. Gao, and Z. Liu, 'Visual-Textual Association with Hardest and Semi-Hard Negative Pairs Mining for Person Search,' Dec. 2019.
- [43] R. Gomez, L. Gomez, J. Gibert, and D. Karatzas, 'Chapter 9 - SelfSupervised Learning from Web Data for Multimodal Retrieval,' in Multimodal Scene Understanding (M. Y. Yang, B. Rosenhahn, and V. Murino, eds.), pp. 279-306, Academic Press, Jan. 2019.
- [44] D. Gray, S. Brennan, and H. Tao, 'Evaluating appearance models for recognition, reacquisition, and tracking,' Proc. IEEE Int. Workshop Vis. Surveill. Perform. Eval. Tracking Surveill., Oct. , Jan. 2007.
- [45] J. Gu, J. Cai, S. Joty, L. Niu, and G. Wang, 'Look, Imagine and Match: Improving Textual-Visual Cross-Modal Retrieval with Generative Models,' June 2018.
- [46] X. Han, L. Yu, X. Zhu, L. Zhang, Y.-Z. Song, and T. Xiang, 'FashionViL: Fashion-Focused Vision-and-Language Representation Learning,' arXiv, 2022.
- [47] X. Han, S. He, L. Zhang, and T. Xiang, 'Text-Based Person Search with Limited Data,' in BMVC 2021 , arXiv, Oct. 2021.
- [48] K. He, X. Zhang, S. Ren, and J. Sun, 'Deep Residual Learning for Image Recognition,' in 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , pp. 770-778, June 2016.
- [49] A. Hermans, L. Beyer, and B. Leibe, 'In Defense of the Triplet Loss for Person Re-Identification,' Nov. 2017.
- [50] N. Huang, J. Liu, Y. Miao, Q. Zhang, and J. Han, 'Deep learning for visible-infrared cross-modality person re-identification: A comprehensive review,' Information Fusion , vol. 91, pp. 396-411, Mar. 2023.
- [51] Z. Huang, Z. Zeng, Y. Huang, B. Liu, D. Fu, and J. Fu, 'Seeing Out of tHe bOx: End-to-End Pre-training for Vision-Language Representation Learning,' 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pp. 12971-12980, June 2021.
- [52] S. Iodice and K. Mikolajczyk, 'Text Attribute Aggregation and Visual Feature Decomposition for Person Search,' in 2020 British Machine Vision Conference , 2020.
- [53] K. Islam, 'Person search: New paradigm of person re-identification: A survey and outlook of recent works,' Image and Vision Computing , vol. 101, p. 103970, Sept. 2020.
- [54] B. Jeong, J. Park, and S. Kwak, 'ASMR: Learning Attribute-Based Person Search with Adaptive Semantic Margin Regularizer,' 2021 IEEE/CVF International Conference on Computer Vision (ICCV) , pp. 11996-12005, Oct. 2021.
- [55] D. Jiang and M. Ye, 'Cross-Modal Implicit Relation Reasoning and Aligning for Text-to-Image Person Retrieval,' in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pp. 2787-2797, 2023.
- [56] C. Jia, Y. Yang, Y. Xia, Y.-T. Chen, Z. Parekh, H. Pham, Q. Le, Y.H. Sung, Z. Li, and T. Duerig, 'Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision,' in Proceedings of the 38th International Conference on Machine Learning , pp. 4904-4916, PMLR, July 2021.
- [57] Z. Ji, J. Hu, D. Liu, L. Y. Wu, and Y. Zhao, 'Asymmetric CrossScale Alignment for Text-Based Person Search,' IEEE Transactions on Multimedia , pp. 1-11, 2022.
- [58] Y. Jing, W. Wang, L. Wang, and T. Tan, 'Cross-Modal Cross-Domain Moment Alignment Network for Person Search,' 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pp. 1067510683, June 2020.
- [59] Y. Jing, C. Si, J. Wang, W. Wang, L. Wang, and T. Tan, 'Pose-Guided Joint Global and Attentive Local Matching Network for Text-Based Person Search,' Sept. 2018.
- [60] Y. Jing, C. Si, J. Wang, W. Wang, L. Wang, and T. Tan, 'Pose-Guided Multi-Granularity Attention Network for Text-Based Person Search,' Proceedings of the AAAI Conference on Artificial Intelligence , vol. 34, pp. 11189-11196, Apr. 2020.

- [61] S. karanam, M. Gou, Z. Wu, A. Rates-Borras, O. Camps, and R. J. Radke, 'A Systematic Evaluation and Benchmark for Person ReIdentification: Features, Metrics, and Datasets,' IEEE Transactions on Pattern Analysis and Machine Intelligence , vol. 41, pp. 523-536, Mar. 2019.
- [62] B. Lavi, I. Ullah, M. Fatan, and A. Rocha, 'Survey on Reliable Deep Learning-Based Person Re-Identification Models: Are We There Yet?,' Apr. 2020.
- [63] K.-H. Lee, X. Chen, G. Hua, H. Hu, and X. He, 'Stacked Cross Attention for Image-Text Matching,' July 2018.
- [64] J. Li, R. R. Selvaraju, A. D. Gotmare, S. R. Joty, C. Xiong, and S. Hoi, 'Align before Fuse: Vision and Language Representation Learning with Momentum Distillation,' in Neural Information Processing Systems , July 2021.
- [65] S. Liao, Y. Hu, Xiangyu Zhu, and S. Z. Li, 'Person re-identification by Local Maximal Occurrence representation and metric learning,' in 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , (Boston, MA, USA), pp. 2197-2206, IEEE, June 2015.
- [66] S. Li, S. Bak, P. Carr, and X. Wang, 'Diversity Regularized Spatiotemporal Attention for Video-based Person Re-identification,' Mar. 2018.
- [67] W. Li, R. Zhao, and X. Wang, 'Human Reidentification with Transferred Metric Learning,' in Computer Vision -ACCV 2012 (K. M. Lee, Y. Matsushita, J. M. Rehg, and Z. Hu, eds.), Lecture Notes in Computer Science, (Berlin, Heidelberg), pp. 31-44, Springer, 2013.
- [68] Y. Li, H. Xu, and J. Xiao, 'Hybrid Attention Network for LanguageBased Person Search,' Sensors , vol. 20, p. 5279, Jan. 2020.
- [69] S. Li, T. Xiao, H. Li, W. Yang, and X. Wang, 'Identity-Aware TextualVisual Matching With Latent Co-Attention,' in Proceedings of the IEEE International Conference on Computer Vision , pp. 1890-1899, 2017.
- [70] S. Li, A. Lu, Y. Huang, C. Li, and L. Wang, 'Joint Token and Feature Alignment Framework for Text-Based Person Search,' IEEE Signal Processing Letters , vol. 29, pp. 2238-2242, 2022.
- [71] S. Li, M. Cao, and M. Zhang, 'Learning Semantic-Aligned Feature Representation for Text-based Person Search,' Dec. 2021.
- [72] Y. Lin, L. Zheng, Z. Zheng, Y. Wu, Z. Hu, C. Yan, and Y. Yang, 'Improving person re-identification by attribute and identity learning,' Pattern Recognition , vol. 95, pp. 151-161, Nov. 2019.
- [73] X. Lin, P. Ren, Y. Xiao, X. Chang, and A. Hauptmann, 'Person search challenges and solutions: A survey,' in Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence , pp. 4500-4507, Association for the Advancement of Artificial Intelligence (AAAI), 2021.
- [74] C. Li, X. Yang, K. Yin, Y. Chang, Z. Wang, and G. Yin, 'Pedestrian re-identification based on attribute mining and reasoning,' IET Image Processing , vol. 15, no. 11, pp. 2399-2411, 2021.
- [75] S. Li, T. Xiao, H. Li, B. Zhou, D. Yue, and X. Wang, 'Person Search With Natural Language Description,' in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pp. 1970-1979, 2017.
- [76] F. Li, H. Zhou, H. Li, Y. Zhang, and Z. Yu, 'Person Text-Image Matching via Text-Feature Interpretability Embedding and External Attack Node Implantation,' 2022.
- [77] D. Li, Z. Zhang, X. Chen, H. Ling, and K. Huang, 'A Richly Annotated Dataset for Pedestrian Attribute Recognition,' Apr. 2016.
- [78] D. Li, Z. Zhang, X. Chen, and K. Huang, 'A Richly Annotated Pedestrian Dataset for Person Retrieval in Real Surveillance Scenarios,' IEEE Transactions on Image Processing , vol. 28, pp. 1575-1590, Apr. 2019.
- [79] H. Li, J. Xiao, M. Sun, E. Lim, and Y. Zhao, 'Transformer-Based Language-Person Search With Multiple Region Slicing,' IEEE Transactions on Circuits and Systems for Video Technology , Apr. 2021.
- [80] T. Li, J. Liu, W. Zhang, Y. Ni, W. Wang, and Z. Li, 'UAV-Human: A Large Benchmark for Human Behavior Understanding with Unmanned Aerial Vehicles,' Aug. 2021.
- [81] J. Liu, Z.-J. Zha, R. Hong, M. Wang, and Y. Zhang, 'Deep Adversarial Graph Attention Convolution Network for Text-Based Person Search,' in Proceedings of the 27th ACM International Conference on Multimedia , MM'19, (New York, NY, USA), pp. 665-673, Association for Computing Machinery, Oct. 2019.
- [82] T.-Y. Liu, C. Zhu, and L. Yang, 'Efficient Text-based Person Search via Single-stage Identity-guided Attribute Parsing and Alignment,' 2022 26th International Conference on Pattern Recognition (ICPR) , pp. 4111-4117, Aug. 2022.
- [83] C. Liu, Z. Mao, T. Zhang, H. Xie, B. Wang, and Y. Zhang, 'Graph Structured Network for Image-Text Matching,' in 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pp. 10918-10927, June 2020.
- [84] X. Liu, H. Zhao, M. Tian, L. Sheng, J. Shao, S. Yi, J. Yan, and X. Wang, 'HydraPlus-Net: Attentive Deep Features for Pedestrian Analysis,' Sept. 2017.
- [85] Z. Liu, Y. Lin, Y. Cao, H. Hu, Y. Wei, Z. Zhang, S. Lin, and B. Guo, 'Swin Transformer: Hierarchical Vision Transformer using Shifted Windows,' Aug. 2021.
- [86] Y. Liu, P. Xiong, L. Xu, S. Cao, and Q. Jin, 'TS2-Net: Token Shift and Selection Transformer for Text-Video Retrieval,' in Computer Vision ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XIV , (Berlin, Heidelberg), pp. 319-335, SpringerVerlag, Oct. 2022.
- [87] C. Luo, Y. Chen, N. Wang, and Z. Zhang, 'Spectral Feature Transformation for Person Re-identification,' Nov. 2018.
- [88] C. Manning, M. Surdeanu, J. Bauer, J. Finkel, S. Bethard, and D. McClosky, 'The Stanford CoreNLP Natural Language Processing Toolkit,' in Proceedings of 52nd Annual Meeting of the Association for Computational Linguistics: System Demonstrations , (Baltimore, Maryland), pp. 55-60, Association for Computational Linguistics, June 2014.
- [89] N. Mathur, S. Mathur, D. Mathur, and P. Dadheech, 'A Brief Survey of Deep Learning Techniques for Person Re-identification,' in 2020 3rd International Conference on Emerging Technologies in Computer Engineering: Machine Learning and Internet of Things (ICETCE) , pp. 129138, Feb. 2020.
- [90] Z. Ming, M. Zhu, X. Wang, J. Zhu, J. Cheng, C. Gao, Y. Yang, and X. Wei, 'Deep learning-based person re-identification methods: A survey and outlook of recent works,' Image and Vision Computing , vol. 119, p. 104394, Mar. 2022.
- [91] B. Ni, H. Peng, M. Chen, S. Zhang, G. Meng, J. Fu, S. Xiang, and H. Ling, 'Expanding Language-Image Pretrained Models for General Video Recognition,' in Computer Vision -ECCV 2022 (S. Avidan, G. Brostow, M. Ciss´ e, G. M. Farinella, and T. Hassner, eds.), Lecture Notes in Computer Science, (Cham), pp. 1-18, Springer Nature Switzerland, 2022.
- [92] K. Niu, L. Huang, Y. Huang, P. Wang, L. Wang, and Y. Zhang, 'Cross-modal Co-occurrence Attributes Alignments for Person Search by Language,' in Proceedings of the 30th ACM International Conference on Multimedia , MM '22, (New York, NY, USA), pp. 4426-4434, Association for Computing Machinery, Oct. 2022.
- [93] K. Niu, Y. Huang, and L. Wang, 'Fusing Two Directions in CrossDomain Adaption for Real Life Person Search by Language,' in 2019 IEEE/CVF International Conference on Computer Vision Workshop (ICCVW) , pp. 1815-1818, Oct. 2019.
- [94] K. Niu, Y. Huang, W. Ouyang, and L. Wang, 'Improving DescriptionBased Person Re-Identification by Multi-Granularity Image-Text Alignments,' IEEE Transactions on Image Processing , vol. 29, pp. 5542-5556, 2020.
- [95] K. Niu, Y. Huang, and L. Wang, 'Textual Dependency Embedding for Person Search by Language,' Proceedings of the 28th ACM International Conference on Multimedia , pp. 4032-4040, Oct. 2020.
- [96] 'Papers with Code -UAV-Human: A Large Benchmark for Human Behavior Understanding with Unmanned Aerial Vehicles.' https://paperswithcode.com/paper/uav-human-a-large-benchmark-forhuman.
- [97] J. Park, B. Jeong, J. Shin, J. Lee, and S. Kwak, 'Learning Discriminative Part Features Through Attentions For Effective And Scalable Person Search,' in 2020 IEEE International Conference on Image Processing (ICIP) , pp. 2376-2380, Oct. 2020.
- [98] Y. Peng, J. Wu, B. Xu, C. Cao, X. Liu, Z. Sun, and Z. He, 'Deep Learning-Based Occluded Person Re-Identification: A Survey,' ACM Transactions on Multimedia Computing, Communications, and Applications , July 2023.
- [99] N. Perwaiz, M. Shahzad, and M. M. Fraz, 'Ubiquitous vision of transformers for person re-identification,' Machine Vision and Applications , vol. 34, p. 27, Feb. 2023.
- [100] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, G. Krueger, and I. Sutskever, 'Learning Transferable Visual Models From Natural Language Supervision,' in Proceedings of the 38th International Conference on Machine Learning , pp. 8748-8763, PMLR, July 2021.
- [101] S. Reed, Z. Akata, B. Schiele, and H. Lee, 'Learning Deep Representations of Fine-grained Visual Descriptions,' May 2016.
- [102] N. Sarafianos, X. Xu, and I. A. Kakadiaris, 'Adversarial Representation Learning for Text-to-Image Matching,' in Proceedings of the IEEE/CVF International Conference on Computer Vision , pp. 5814-5824, 2019.
- [103] W. J. Scheirer, N. Kumar, P. N. Belhumeur, and T. E. Boult, 'Multiattribute spaces: Calibration for attribute fusion and similarity search,'

- in 2012 IEEE Conference on Computer Vision and Pattern Recognition , pp. 2933-2940, June 2012.
- [104] M. Schuster and K. Paliwal, 'Bidirectional recurrent neural networks,' IEEE Transactions on Signal Processing , vol. 45, pp. 2673-2681, Nov. 1997.
- [105] A. Shah and T. Vuong, 'Natural Language Person Search Using Deep Reinforcement Learning,' Sept. 2018.
- [106] Z. Shao, X. Zhang, M. Fang, Z. Lin, J. Wang, and C. Ding, 'Learning Granularity-Unified Representations for Text-to-Image Person Reidentification,' in Proceedings of the 30th ACM International Conference on Multimedia , (Lisboa Portugal), pp. 5566-5574, ACM, Oct. 2022.
- [107] X. Shu, W. Wen, H. Wu, K. Chen, Y. Song, R. Qiao, B. Ren, and X. Wang, 'See Finer, See More: Implicit Modality Alignment for TextBased Person Retrieval,' in Computer Vision - ECCV 2022 Workshops (L. Karlinsky, T. Michaeli, and K. Nishino, eds.), Lecture Notes in Computer Science, (Cham), pp. 624-641, Springer Nature Switzerland, 2023.
- [108] N. K. Singh, M. Khare, and H. B. Jethva, 'A comprehensive survey on person re-identification approaches: Various aspects,' Multimedia Tools and Applications , vol. 81, pp. 15747-15791, May 2022.
- [109] H. O. Song, Y. Xiang, S. Jegelka, and S. Savarese, 'Deep Metric Learning via Lifted Structured Feature Embedding,' Nov. 2015.
- [110] Y. Sun, L. Zheng, Y. Yang, Q. Tian, and S. Wang, 'Beyond Part Models: Person Retrieval with Refined Part Pooling (and A Strong Convolutional Baseline),' in Proceedings of the European Conference on Computer Vision (ECCV) , pp. 480-496, 2018.
- [111] Y. Sun, L. Zheng, W. Deng, and S. Wang, 'SVDNet for Pedestrian Retrieval,' Aug. 2017.
- [112] W. Suo, M. Sun, K. Niu, Y. Gao, P. Wang, Y. Zhang, and Q. Wu, 'A Simple and Robust Correlation Filtering Method for Text-Based Person Search,' in Computer Vision - ECCV 2022 - 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXXV (S. Avidan, G. J. Brostow, M. Ciss´ e, G. M. Farinella, and T. Hassner, eds.), vol. 13695 of Lecture Notes in Computer Science , pp. 726-742, Springer, 2022.
- [113] C. Tang, L. Sheng, Z. Zhang, and X. Hu, 'Improving Pedestrian Attribute Recognition With Weakly-Supervised Multi-Scale AttributeSpecific Localization,' Oct. 2019.
- [114] D. A. Vaquero, R. S. Feris, D. Tran, L. Brown, A. Hampapur, and M. Turk, 'Attribute-based people search in surveillance environments,' in 2009 Workshop on Applications of Computer Vision (WACV) , pp. 1-8, Dec. 2009.
- [115] R. R. Varior, B. Shuai, J. Lu, D. Xu, and G. Wang, 'A Siamese Long Short-Term Memory Architecture for Human Re-Identification,' July 2016.
- [116] O. Vinyals, A. Toshev, S. Bengio, and D. Erhan, 'Show and Tell: A Neural Image Caption Generator,' in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pp. 3156-3164, 2015.
- [117] Z. Wang, J. Xue, A. Zhu, Y. Li, M. Zhang, and C. Zhong, 'AMEN: Adversarial Multi-space Embedding Network for Text-Based Person Reidentification,' in Pattern Recognition and Computer Vision: 4th Chinese Conference, PRCV 2021, Beijing, China, October 29 - November 1, 2021, Proceedings, Part II , (Berlin, Heidelberg), pp. 462-473, Springer-Verlag, Oct. 2021.
- [118] Z. Wang, A. Zhu, J. Xue, X. Wan, C. Liu, T. Wang, and Y. Li, 'CAIBC: Capturing All-round Information Beyond Color for Text-based Person Retrieval,' in Proceedings of the 30th ACM International Conference on Multimedia , MM '22, (New York, NY, USA), pp. 5314-5322, Association for Computing Machinery, Oct. 2022.
- [119] Y. Wang, S. Yang, S. Liu, and Z. Zhang, 'Cross-Domain Person Reidentification: A Review,' in Artificial Intelligence in China (Q. Liang, W. Wang, J. Mu, X. Liu, Z. Na, and X. Cai, eds.), Lecture Notes in Electrical Engineering, (Singapore), pp. 153-160, Springer, 2021.
- [120] C. Wang, Z. Luo, Z. Zhong, and S. Li, 'Divide-and-Merge the embedding space for cross-modality person search,' Neurocomputing , vol. 463, pp. 388-399, Nov. 2021.
- [121] G. Wang, F. Yu, J. Li, Q. Jia, and S. Ding, 'Exploiting the Textual Potential from Vision-Language Pre-training for Text-based Person Search,' Mar. 2023.
- [122] W. Wang, H. Bao, L. Dong, J. Bjorck, Z. Peng, Q. Liu, K. Aggarwal, O. K. Mohammed, S. Singhal, S. Som, and F. Wei, 'Image as a Foreign Language: BEiT Pretraining for All Vision and Vision-Language Tasks,' Aug. 2022.
- [123] Z. Wang, A. Zhu, Z. Zheng, J. Jin, Z. Xue, and G. Hua, 'IMG-Net: Inner-cross-modal attentional multigranular network for description-based person re-identification,' Journal of Electronic Imaging , vol. 29, Aug. 2020.
- [124] C. Wang, Z. Luo, Y. Lin, and S. Li, 'Improving embedding learning by virtual attribute decoupling for text-based person search,' Neural Computing and Applications , vol. 34, pp. 5625-5647, Apr. 2022.
- [125] Z. Wang, C. Li, A. Zheng, R. He, and J. Tang, 'Interact, Embed, and EnlargE: Boosting Modality-Specific Representations for MultiModal Person Re-identification,' Proceedings of the AAAI Conference on Artificial Intelligence , vol. 36, pp. 2633-2641, June 2022.
- [126] Z. Wang, Z. Wang, Y. Zheng, Y. Wu, W. Zeng, and S. Satoh, 'Beyond Intra-modality: A Survey of Heterogeneous Person Re-identification,' Apr. 2020.
- [127] Y. Wang, C. Bo, D. Wang, S. Wang, Y. Qi, and H. Lu, 'Language Person Search with Mutually Connected Classification Loss,' in ICASSP 2019 - 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pp. 2057-2061, May 2019.
- [128] J. Wang, Y. Song, T. Leung, C. Rosenberg, J. Wang, J. Philbin, B. Chen, and Y. Wu, 'Learning Fine-Grained Image Similarity with Deep Ranking,' in 2014 IEEE Conference on Computer Vision and Pattern Recognition , (Columbus, OH, USA), pp. 1386-1393, IEEE, June 2014.
- [129] Z. Wang, A. Zhu, J. Xue, X. Wan, C. Liu, T. Wang, and Y. Li, 'Look Before You Leap: Improving Text-based Person Retrieval by Learning A Consistent Cross-modal Common Manifold,' Proceedings of the 30th ACM International Conference on Multimedia , pp. 1984-1992, Oct. 2022.
- [130] Y. Wang, D. Qi, and C. Zhao, 'Part-Based Multi-Scale Attention Network for Text-Based Person Search,' in Pattern Recognition and Computer Vision (S. Yu, Z. Zhang, P. C. Yuen, J. Han, T. Tan, Y. Guo, J. Lai, and J. Zhang, eds.), Lecture Notes in Computer Science, (Cham), pp. 462-474, Springer International Publishing, 2022.
- [131] Y. Wang, Z. Chen, F. Wu, and G. Wang, 'Person Re-Identification With Cascaded Pairwise Convolutions,' in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pp. 1470-1478, 2018.
- [132] T. Wang, H. Liu, P. Song, T. Guo, and W. Shi, 'Pose-Guided Feature Disentangling for Occluded Person Re-identification Based on Transformer,' Proceedings of the AAAI Conference on Artificial Intelligence , vol. 36, pp. 2540-2549, June 2022.
- [133] Z. Wang, A. Zhu, J. Xue, D. Jiang, C. Liu, Y. Li, and F. Hu, 'SUM: Serialized Updating and Matching for text-based person retrieval,' Knowledge-Based Systems , vol. 248, p. 108891, July 2022.
- [134] K. Wang, H. Wang, M. Liu, X. Xing, and T. Han, 'Survey on person reidentification based on deep learning,' CAAI Transactions on Intelligence Technology , vol. 3, no. 4, pp. 219-227, 2018.
- [135] C. Wang, Z. Luo, Y. Lin, and S. Li, 'Text-based Person Search via Multi-Granularity Embedding Learning,' in Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence , (Montreal, Canada), pp. 1068-1074, International Joint Conferences on Artificial Intelligence Organization, Aug. 2021.
- [136] Z. Wang, Z. Fang, J. Wang, and Y. Yang, 'ViTAA: Visual-Textual Attributes Alignment in Person Search by Natural Language,' in Computer Vision - ECCV 2020 (A. Vedaldi, H. Bischof, T. Brox, and J.-M. Frahm, eds.), Lecture Notes in Computer Science, (Cham), pp. 402-420, Springer International Publishing, 2020.
- [137] X.-S. Wei, Y.-Z. Song, O. Mac Aodha, J. Wu, Y. Peng, J. Tang, J. Yang, and S. Belongie, 'Fine-Grained Image Analysis with Deep Learning: A Survey,' Nov. 2021.
- [138] L. Wei, S. Zhang, W. Gao, and Q. Tian, 'Person Transfer GAN to Bridge Domain Gap for Person Re-identification,' in 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition , pp. 79-88, June 2018.
- [139] D. Wu, S.-J. Zheng, X.-P. Zhang, C.-A. Yuan, F. Cheng, Y. Zhao, Y.-J. Lin, Z.-Q. Zhao, Y.-L. Jiang, and D.-S. Huang, 'Deep learningbased methods for person re-identification: A comprehensive review,' Neurocomputing , vol. 337, pp. 354-371, Apr. 2019.
- [140] Y. Wu, Z. Yan, X. Han, G. Li, C. Zou, and S. Cui, 'LapsCore: Language-Guided Person Search via Color Reasoning,' in Proceedings of the IEEE/CVF International Conference on Computer Vision , pp. 16241633, 2021.
- [141] D. Wu, H. Huang, Q. Zhao, S. Zhang, J. Qi, and J. Hu, 'Overview of deep learning based pedestrian attribute recognition and re-identification,' Heliyon , vol. 8, p. e12086, Dec. 2022.
- [142] T. Xiao, S. Li, B. Wang, L. Lin, and X. Wang, 'Joint Detection and Identification Feature Learning for Person Search,' Apr. 2016.
- [143] T. Xiao, S. Li, B. Wang, L. Lin, and X. Wang, 'Joint Detection and Identification Feature Learning for Person Search,' in 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , pp. 33763385, July 2017.
- [144] H. Xu, M. Yan, C. Li, B. Bi, S. Huang, W. Xiao, and F. Huang, 'E2EVLP: End-to-End Vision-Language Pre-training Enhanced by Visual Learning,' Proceedings of the 59th Annual Meeting of the Association for

- Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) , pp. 503-513, 2021.
- [145] H. Xu, K. He, B. A. Plummer, L. Sigal, S. Sclaroff, and K. Saenko, 'Multilevel Language and Vision Integration for Text-to-Clip Retrieval,' Proceedings of the AAAI Conference on Artificial Intelligence , vol. 33, pp. 9062-9069, July 2019.
- [146] E. Yaghoubi, A. Kumar, and H. Proenc ¸a, 'SSS-PR: A short survey of surveys in person re-identification,' Pattern Recognition Letters , vol. 143, pp. 50-57, Mar. 2021.
- [147] S. Yan, N. Dong, L. Zhang, and J. Tang, 'CLIP-Driven Fine-grained Text-Image Person Re-identification,' 2022.
- [148] S. Yan, Y. Zhang, M. Xie, D. Zhang, and Z. Yu, 'Cross-domain person re-identification with pose-invariant feature decomposition and hypergraph structure alignment,' Neurocomputing , vol. 467, pp. 229-241, Jan. 2022.
- [149] C. Yang, F. Qi, and H. Jia, 'Survey on Unsupervised Techniques for Person Re-Identification,' in 2021 2nd International Conference on Computing and Data Science (CDS) , pp. 161-164, Jan. 2021.
- [150] S. Yang, Y. Zhou, Y. Wang, Y. Wu, L. Zhu, and Z. Zheng, 'Towards Unified Text-based Person Retrieval: A Large-scale Multi-Attribute and Language Search Benchmark,' Aug. 2023.
- [151] S. Yan, H. Tang, L. Zhang, and J. Tang, 'Image-Specific Information Suppression and Implicit Local Alignment for Text-based Person Search,' 2022.
- [152] S. Yan, H. Tang, L. Zhang, and J. Tang, 'Image-Specific Information Suppression and Implicit Local Alignment for Text-based Person Search,' July 2023.
- [153] F. Yan, J. Kittler, and K. Mikolajczyk, 'Person Re-Identification with Vision and Language,' in 2018 24th International Conference on Pattern Recognition (ICPR) , pp. 2136-2141, Aug. 2018.
- [154] M. Ye, J. Shen, G. Lin, T. Xiang, L. Shao, and S. C. H. Hoi, 'Deep Learning for Person Re-Identification: A Survey and Outlook,' IEEE Transactions on Pattern Analysis and Machine Intelligence , vol. 44, pp. 2872-2893, June 2022.
- [155] M. Ye, C. Liang, Z. Wang, Q. Leng, J. Chen, and J. Liu, 'Specific Person Retrieval via Incomplete Text Description,' in Proceedings of the 5th ACM on International Conference on Multimedia Retrieval , ICMR '15, (New York, NY, USA), pp. 547-550, Association for Computing Machinery, June 2015.
- [156] M. Ye, Z. Wang, X. Lan, and P. C. Yuen, 'Visible Thermal Person ReIdentification via Dual-Constrained Top-Ranking,' in Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence , (Stockholm, Sweden), pp. 1092-1099, International Joint Conferences on Artificial Intelligence Organization, July 2018.
- [157] Z. Yin, W.-S. Zheng, A. Wu, H.-X. Yu, H. Wan, X. Guo, F. Huang, and J. Lai, 'Adversarial attribute-image person re-identification,' in Proceedings of the 27th International Joint Conference on Artificial Intelligence , IJCAI'18, (Stockholm, Sweden), pp. 1100-1106, AAAI Press, July 2018.
- [158] R. Yu, D. Du, R. LaLonde, D. Davila, C. Funk, A. Hoogs, and B. Clipp, 'Cascade Transformers for End-to-End Person Search,' Mar. 2022.
- [159] A. Zahra, N. Perwaiz, M. Shahzad, and M. M. Fraz, 'Person reidentification: A retrospective on domain specific open challenges and future trends,' Pattern Recognition , vol. 142, p. 109669, Oct. 2023.
- [160] R. Zellers, Y. Bisk, A. Farhadi, and Y. Choi, 'From Recognition to Cognition: Visual Commonsense Reasoning,' in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pp. 6720-6731, 2019.
- [161] Y. Zhang and H. Lu, 'Deep Cross-Modal Projection Learning for Image-Text Matching,' in Computer Vision - ECCV 2018 (V. Ferrari, M. Hebert, C. Sminchisescu, and Y. Weiss, eds.), Lecture Notes in Computer Science, (Cham), pp. 707-723, Springer International Publishing, 2018.
- [162] H. Zhang, P. Zhang, X. Hu, Y.-C. Chen, L. H. Li, X. Dai, L. Wang, L. Yuan, J.-N. Hwang, and J. Gao, 'GLIPv2: Unifying Localization and Vision-Language Understanding,' in Advances in Neural Information Processing Systems , Oct. 2022.
- [163] L. Zhang, G. Du, F. Liu, H. Tu, and X. Shu, 'Global-Local Multiple Granularity Learning for Cross-Modality Visible-Infrared Person Reidentification,' IEEE Transactions on Neural Networks and Learning Systems , pp. 1-11, 2021.
- [164] S. Zhang, D. Long, Y. Gao, L. Gao, Q. Zhang, K. Niu, and Y. Zhang, 'Text-based Person Search in Full Images via Semantic-Driven Proposal Generation,' ArXiv , Sept. 2021.
- [165] S. Zhao, C. Gao, Y. Shao, W.-S. Zheng, and N. Sang, 'Weakly Supervised Text-Based Person Re-Identification,' in Proceedings of the
- IEEE/CVF International Conference on Computer Vision , pp. 1139511404, 2021.
- [166] Z. Zheng, L. Zheng, and Y. Yang, 'A Discriminatively Learned CNN Embedding for Person Re-identification,' ACM Transactions on Multimedia Computing, Communications, and Applications , vol. 14, pp. 1-20, Feb. 2018.
- [167] Z. Zheng, L. Zheng, M. Garrett, Y. Yang, M. Xu, and Y.-D. Shen, 'Dual-path Convolutional Image-Text Embeddings with Instance Loss,' ACM Transactions on Multimedia Computing, Communications, and Applications , vol. 16, pp. 51:1-51:23, May 2020.
- [168] K. Zheng, W. Liu, J. Liu, Z.-J. Zha, and T. Mei, 'Hierarchical Gumbel Attention Network for Text-based Person Search,' Proceedings of the 28th ACM International Conference on Multimedia , pp. 3441-3449, Oct. 2020.
- [169] L. Zheng, L. Shen, L. Tian, S. Wang, J. Bu, and Q. Tian, 'Person Re-identification Meets Image Search,' Feb. 2015.
- [170] L. Zheng, Y. Yang, and A. G. Hauptmann, 'Person Re-identification: Past, Present and Future,' Oct. 2016.
- [171] M. Zheng, S. Karanam, Z. Wu, and R. J. Radke, 'Re-Identification With Consistent Attentive Siamese Networks,' in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pp. 5735-5744, 2019.
- [172] T. Zhou, M. Chen, J. Yu, and D. Terzopoulos, 'Attention-Based Natural Language Person Retrieval,' in 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) , pp. 27-34, July 2017.
- [173] A. Zhu, Z. Wang, Y. Li, X. Wan, J. Jin, T. Wang, F. Hu, and G. Hua, 'DSSL: Deep Surroundings-person Separation Learning for Textbased Person Retrieval,' in Proceedings of the 29th ACM International Conference on Multimedia , MM '21, (New York, NY, USA), pp. 209217, Association for Computing Machinery, Oct. 2021.
- [174] H. Zhu, W. Ke, D. Li, J. Liu, L. Tian, and Y. Shan, 'Dual CrossAttention Learning for Fine-Grained Visual Categorization and Object Re-Identification,' in 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pp. 4682-4692, June 2022.
- [175] K. Zhu, H. Guo, Z. Liu, M. Tang, and J. Wang, 'Identity-Guided Human Semantic Parsing for Person Re-identification,' Computer Vision - ECCV 2020 , vol. 12348, pp. 346-363, 2020.
- [176] G. Zou, G. Fu, X. Peng, Y. Liu, M. Gao, and Z. Liu, 'Person reidentification based on metric learning: A survey,' Multimedia Tools and Applications , vol. 80, pp. 26855-26888, July 2021.
- [177] Y. DENG, P. Luo, C. C. Loy, and X. Tang, 'Pedestrian Attribute Recognition At Far Distance,' in Proceedings of the 22nd ACM International Conference on Multimedia , MM '14, (New York, NY, USA), pp. 789792, Association for Computing Machinery, Nov. 2014.
- [178] M. Hirzer, C. Beleznai, P. M. Roth, and H. Bischof, 'Person reidentification by descriptive and discriminative classification,' in Proceedings of the 17th Scandinavian Conference on Image Analysis , SCIA'11, (Berlin, Heidelberg), pp. 91-102, Springer-Verlag, May 2011.
- [179] R. Layne, T. M. Hospedales, and S. Gong, 'Person Re-identification by Attributes,' in British Machine Vision Conference, BMVC 2012, Surrey, UK, September 3-7, 2012 , pp. 1-11, BMVA Press, 2012.
- [180] W. Li, R. Zhao, T. Xiao, and X. Wang, 'DeepReID: Deep Filter Pairing Neural Network for Person Re-identification,' in Proceedings of the 2014 IEEE Conference on Computer Vision and Pattern Recognition , CVPR '14, (USA), pp. 152-159, IEEE Computer Society, June 2014.
- [181] Y. Lin, L. Zheng, Z. Zheng, Y. Wu, Z. Hu, C. Yan, and Y. Yang, 'Improving person re-identification by attribute and identity learning,' Pattern Recognition , vol. 95, pp. 151-161, Nov. 2019.
- [182] X. Wang, S. Zheng, R. Yang, B. Luo, and J. Tang, 'Pedestrian Attribute Recognition: A Survey,' Jan. 2019.
- [183] F. Xiong, M. Gou, O. Camps, and M. Sznaier, 'Person Re-Identification Using Kernel-Based Metric Learning Methods,' in Computer Vision ECCV 2014 (D. Fleet, T. Pajdla, B. Schiele, and T. Tuytelaars, eds.), (Cham), pp. 1-16, Springer International Publishing, 2014.
- [184] Z. Cao, G. Hidalgo, T. Simon, S.-E. Wei, and Y. Sheikh, 'OpenPose: Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields,' May 2019.
- [185] X. Pan, P. Luo, J. Shi, and X. Tang, 'Two at Once: Enhancing Learning and Generalization Capacities via IBN-Net,' Mar. 2020.
- [186] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, G. Krueger, and I. Sutskever, 'Learning Transferable Visual Models From Natural Language Supervision,' Feb. 2021.
- [187] R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer, 'HighResolution Image Synthesis with Latent Diffusion Models,' Apr. 2022.

2014

Fig. 6: The figure shows the development of the main categorical dimensions and surveys of Person Re-ID throughout history. Overall, the large figure on the left represents the relevance of influential Person Re-ID surveys and their categorical dimensions, with the line font implicitly indicating their phase weights. The smaller figure on the top right is a more visual representation of the temporal relationships between the emergence of these Person Re-ID surveys.

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00096v1-figure-e40a1abd-a7b8-4cd9-80a0-e070912be51a.jpg)

- [188] O. Ronneberger, P. Fischer, and T. Brox, 'U-Net: Convolutional Networks for Biomedical Image Segmentation,' May 2015.
- [189] L. Zhang, A. Rao, and M. Agrawala, 'Adding conditional control to text-to-image diffusion models,' arXiv preprint arXiv:2302.05543 , 2023. [190] T. B. Brown, 'Language models are few-shot learners,' arXiv preprint arXiv:2005.14165 , Jul 2020.

and further support the findings discussed. They are referenced throughout the paper where relevant.

- [191] A. Hertz, R. Mokady, J. Tenenbaum, K. Aberman, Y. Pritch, and D. Cohen-Or, 'Prompt-to-prompt image editing with cross attention control,' arXiv preprint arXiv:2208.01626 , Aug 2022.
- [192] S. Zhang, Y. Wang, T. Chai, A. Li, and A. K. Jain, 'Realgait: Gait recognition for person re-identification,' arXiv preprint arXiv:2201.04806 , 2022.

## APPENDIX

This appendix provides tables and figures that supplement the data and analyses presented in the main body of the paper. These tables and figures provide more detailed information

Fig. 7: The figure shows a tree development diagram of all existing text-based pedestrian re-recognition methods. In the vertical direction from bottom to top represents the sequence of time development, and in the horizontal direction different trunk colors represent different trunk methods. Green represents Graph/GAN-based methods, pink represents ResNet/BiLSTMbased methods, blue represents ViT/BERT-based methods, orange represents CLIP multimodal macromodel-based methods, and grey represents text-based pedestrian re-recognition research prior to its emergence, which also appears as a comparison method.

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00096v1-figure-3c656da0-bb72-47f8-99ac-82f4fc2943cf.jpg)

TABLE III: The table compares the aspects emphasized by existing surveys based on chronological order in terms of the dimensions of authors, journal name, and major contributions.

| Author               | Publication                                                                                              | Reference   | Contribution                                                                                                                                                                                                                   |
|----------------------|----------------------------------------------------------------------------------------------------------|-------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| Bedagkar-Gala et al. | A survey of approaches and trends in person re- identification [4]                                       | IVC14       | Introduced person Re-ID challenges up to 2014 and summarizes the mainstream solutions.                                                                                                                                         |
| Zheng et al.         | Person Re-identification Past, Present and Future [170]                                                  | arXiv16     | Discussed the past development mainly in terms of pedestrian retrieval, and image classification. They investigated the image/video-based systems and methods.                                                                 |
| Chahar et al.        | A study on deep convolutional neural network based approaches for person re-identification [9]           | PRMI17      | Presented the methods for pedestrian Re-ID based on CNNs and explained some of the problems and directions for development.                                                                                                    |
| Wang et al.          | Survey on person re-identification based on deep learning [134]                                          | CAAI18      | Surveyed deep learning-based pedestrian re- recognition approaches, and pointed out the ways forward at that time.                                                                                                             |
| Karanam et al.       | A systematic evaluation and benchmark for person re-identification: Features, metrics, and datasets [61] | TPAMI19     | Conducted a comprehensive review and perfor- mance evaluation of single- and multi-shot Re- ID algorithms. Person Re-ID was first started to be introduced in terms of feature extraction and metric learning.                 |
| Wu et al.            | Deep learning-based methods for person re- identification: A comprehensive review [139]                  | NC19        | Summarized six deep learning-based methods for person re-identification, including recognition, verification, distance metrics, parts, video, and data augmentation.                                                           |
| Mathur et al.        | A Brief Survey of Deep Learning Techniques for Person Re-identification [89]                             | TCSVT20     | Described for the first time the disparity of Person Re-ID between closed and open world scenarios and pointed out the limitations of open Re-ID works.                                                                        |
| Islam                | Person search: New paradigm of person re- identification: A survey and outlook of recent works [53]      | IVC20       | Discussed about feature representation learning and deep metric learning with novel loss func- tions.                                                                                                                          |
| Lavi et al.          | Survey on Reliable Deep Learning-Based Person Re-Identification Models: Are We There Yet? [62]           | arXiv20     | Investigated the SOTA DNN models now avail- able for the topic of person re-identification and considered the restrictions of these models.                                                                                    |
| Wang et al.          | Beyond Intra-modality: A Survey of Heteroge- neous Person Re-identification [126]                        | IJCAI20     | Categorized four cross-modal application scenar- ios in Person Re-ID: sketch, text, low resolu- tion(LR)and infrared(IR).                                                                                                      |
| Zou et al.           | Person re-identification based on metric learning: a survey [176]                                        | MTA21       | Summarized the research progress of person Re- ID methods based on metric learning.                                                                                                                                            |
| Ye et al.            | Deep Learning for Person Re-identification: A Survey and Outlook [154]                                   | TPAMI21     | Reviewed for closed-world person Re-ID from three different perspectives, including deep feature representation learning, deep metric learning and ranking optimization                                                        |
| Yang et al.          | Survey on Unsupervised Techniques for Person Re-Identification [149]                                     | CDS21       | Surveyed the SOTA unsupervised Approaches of person Re-ID.                                                                                                                                                                     |
| Yaghoubi et al.      | SSS-PR: A short survey of surveys in person re- identification [146]                                     | PRL21       | Presented a multi-dimensional taxonomy to clas- sify the most pertinent person Re-ID studies based on diverse viewpoints.                                                                                                      |
| Wang et al.          | Cross-Domain Person Re-identification: AReview [119]                                                     | AIC21       | Investigated the available datasets for cross- domain person re-identification and compared the results of existing methods these on the datasets.                                                                             |
| Ming et al.          | Deep learning-based person re-identification methods: A survey and outlook of recent works [90]          | IVC22       | Suggested to divide deep learning based person re- identification approaches to four groups, including deep metric learning, local feature learning, gen- erative adversarial learning, and sequence feature learning methods. |
| Peng et al.          | Deep Learning-Based Occluded Person Re- Identification: A Survey [98]                                    | TMCC22      | Proposed a deep learning -based survey of oc- cluded person re-identification methods.                                                                                                                                         |
| Singh et al.         | A comprehensive survey on person re- identification approaches: various aspects [108]                    | MTA22       | Discussed person re-identification approaches based on image/video data in terms of temporal, spatial, metric, and automation dimensions.                                                                                      |
| Huang et al.         | Deep learning for visible-infrared cross-modality person re-identification: A comprehensive review [50]  | IF22        | Provided a comprehensive classification for those SOTA visible-infrared cross-modality person re- identification models.                                                                                                       |
| Zahra et al.         | Person re-identification: A retrospective on do- main specific open challenges and future trends [159]   | PR23        | Overviewed the work on image/video-based per- son re-identification in four aspects: dataset, ar- chitecture, number of papers, and challenges.                                                                                |

## TABLE IV: Text-based person Re-ID SOTA methods

|                       |                  |                |                     |                          |                           |                                                                               |                                                             |             | CUHK-PEDES              |                   |             | ICFG-PEDES   |                   | RSTPReid    |             |             |
|-----------------------|------------------|----------------|---------------------|--------------------------|---------------------------|-------------------------------------------------------------------------------|-------------------------------------------------------------|-------------|-------------------------|-------------------|-------------|--------------|-------------------|-------------|-------------|-------------|
| Method                | Publication      | Text-to-Image  | Feature             | Visual backbone          | Language backbone         | Auxiliary strategy                                                            | Alignment loss                                              | Rank@1      | Rank@5                  | Rank@10           | Rank@1      | Rank@5       | Rank@10 Rank@1    | Rank@5      | Rank@10     | Rank@10     |
| CNN-RNN[46]           | CVPR16           | TI             | global              | VGG-16                   | Bi-LSTM                   | /                                                                             | ID Loss ID loss                                             | 8.07        | 32.47                   |                   |             |              |                   |             |             |             |
| Neural Talk[42]       | CVPR15           | TI             | global              | VGG-16                   | Bi-LSTM                   | /                                                                             |                                                             | 13.66       |                         | 41.72             |             |              |                   |             |             |             |
| GNA-RNN[4] IATV[17]   | CVPR17 ICCV17    | TI TI          | global global       | VGG-16 VGG-16            | Bi-LSTM LSTM              | Attention Spatial Attention                                                   | ID Loss 19.05 CMCE loss                                     | 25.94       | 60.48                   | 53.64             |             |              |                   |             |             |             |
| PWM-ATH[20]           | WACV18           | TI             | global              | VGG-16                   | Bi-LSTM                   | Attention                                                                     | ID Loss 27.14                                               | 49.45       |                         | 61.02             |             |              |                   |             |             |             |
| GLA[16]               | ECCV18           | TI             | Multi-scale         | ResNet-50                | Bi-LSTM                   | Parts, Mask                                                                   | ID loss                                                     | 43.58       | 66.93                   | 76.26             |             |              |                   |             |             |             |
| CMPM-CMPC[18]         | ECCV18           | IT, TI         | global              | MobileNet                | Bi-LSTM                   | /                                                                             | CMPM-CMPC loss                                              | 49.37       | 71.69                   | 79.27             | 43.51 65.44 |              |                   |             |             |             |
| SCAN[47]              | ECCV18           | TI, IT         | Multi-scale         | ResNet-101, Faster R-CNN | Bi-GRU                    | Region/Attention                                                              | Triplet loss                                                | 55.86       | 75.97                   | 83.69             | 50.05 69.65 |              |                   |             |             |             |
| MCCL[48]              | ICASSP19         | TI             | global              | MobileNet                | Bi-LSTM                   | Text map Attention KL                                                         | divergence, Triplet loss                                    | 50.58       |                         | 79.06             |             |              |                   |             |             |             |
| A-GANet[22]           | MM19             | TI             | Local               | ResNet50/GAC             | Bi-LSTM/GAC               | Region, GAN, GCN                                                              | CMPM loss 53.14                                             | 74.03       | 81.95                   |                   |             |              |                   |             |             |             |
| TiMAM[21]             | ICCV19           | IT, TI         | global              | ResNet-101               | BERT/LSTM                 | GAN binary                                                                    | cross-entropy loss                                          | 54.51       | 77.56                   | 84.78             |             |              |                   |             |             |             |
| CBA[49]               | ICCVW19          | TI global      |                     | ResNet-101               | Bi-GRU                    | Attention                                                                     | ID loss, triplet Loss                                       | 57.84       | 78.33                   |                   |             |              |                   |             |             |             |
| Dual Path[50]         | TOMM20           | TI global      |                     | ResNet-50                | ResNet-50                 | Data distribution                                                             | Instance loss, ranking loss 44.40                           |             | 66.26                   | 85.43 75.07 38.99 | 59.44       |              |                   |             |             |             |
| MIA[23]               | TIP20            | TI             | Multi-scale         | VGG-16                   | Bi-LSTM                   | Region binary                                                                 | cross-entropy loss 53.10                                    | 75.00       | 82.90                   | 46.49             | 67.14       |              |                   |             |             |             |
| GALM PMA[51]          | AAAI20           | TI             | global              | ResNet-50                | Bi-LSTM                   | Pose Estimator ID loss,                                                       | Rank loss 53.81                                             | 73.54       | 81.23                   | 84.56             |             |              |                   |             |             |             |
| TDE[52]               | MM20             |                | global              | ResNet-101               | BERT                      | Attention                                                                     | CMPM loss contrastive                                       | 55.25       | 77.46 75.84             | 83.52             |             |              |                   |             |             |             |
| ViTAA[14]             | ECCV20           | TI TI          | Multi-scale         | ResNet-50                | Bi-LSTM                   | Attribute                                                                     | loss                                                        | 55.97       |                         | 50.98             | 68.79       | 75.78        |                   |             |             |             |
| IMG-Net[53]           | JEI20            | TI             | Multi-scale         | ResNet-50                | BERT                      | parts ID loss,                                                                | triplet loss 56.48                                          | 76.89       | 85.01                   |                   |             |              | 37.60             |             | 61.15       | 61.15       |
| CMAAM[54]             | WACV20           | TI             | Multi-scale         | MobileNet                | Bi-LSTM                   | Attribute ID loss,                                                            | triplet loss, attribute loss                                | 56.68       | 77.18 84.86             |                   |             |              |                   |             |             |             |
| HGAN[15]              | MM20             | IT, TI         | Multi-scale         | ResNet-50                | BERT                      | Parts, NLTK ID loss,                                                          | triplet loss, pair-wise loss                                | 59.00       | 79.49 86.60             |                   |             |              |                   |             |             |             |
| CA[55] CMKA[56]       | Sensors20 TIP21  | TI TI, IT      | global global       | ResNet-50                | Bi-LSTM                   | cubic attention ID / KL                                                       | loss, Ranking loss 60.73 divergence, ID Loss                | 54.69       | 78.63 84.96 73.65       | 81.86             |             |              |                   |             |             |             |
| ITMeetsAL[57]         | PR21             | TI, IT         | global              | ResNet50 ResNet-152      | Bi-LSTM Bi-LSTM           | Shannon theory and adversarial learning ID/triplet                            | loss, KL divergence 55.72                                   | 76.15       | 84.26                   |                   |             |              |                   |             |             |             |
| DME[58]               | Neurocomuting21  |                | Multi-scale         | ResNet50                 | Bi-LSTM                   | Self-attention, part ID Loss,                                                 | CMM loss, RSP loss, CMR loss 56.32                          | 77.23       | 84.71                   |                   |             |              |                   |             |             |             |
| CMMT[59]              | ICCV21           | TI TI          | global              | ResNet-50                | Bi-LSTM                   | Clustering                                                                    | CMPM Loss 57.10                                             |             | 78.14 85.23             |                   |             |              |                   |             |             |             |
| T-MRS[60]             | TCSVT21          | TI             | Multi-scale         | ResNet-101               | BERT                      | Vanilla / overlapped/ keypoints slicing                                       | ID/contrastive loss 57.67                                   | 78.25       | 84.93                   |                   |             |              |                   |             |             |             |
| TBPS[61]              | BMVC21           | TI, IT         | global              | CLIP/ResNet-101          | CLIP-TE, Bi-GRU           | CLIP                                                                          | contrastive loss, ID loss 64.08                             |             | 81.73                   | 88.19             |             |              |                   |             |             |             |
| NAFS[62]              | arXiv21          | TI             | Adaptive Full-scale | ResNet-50                | BERT-Base-Uncased         | Parts,contextual non-local attention CMPM/CSAL ID/triplet/ranking/adversarial | loss                                                        | 59.94       | 79.86 86.22             | 86.70             |             |              | 38.45             |             |             |             |
| AMEN[63]              | PRCV21           | TI             | global              | ResNet-50                | Bi-GRU                    | Graph, GAN, Autoencoder                                                       | loss 57.16                                                  |             | 78.64                   |                   |             |              |                   |             | 62.40       | 62.40       |
| MGEL[64]              | IJCAI21          | TI             | Multi-scale         | ResNet-50                | Bi-LSTM                   | Multi-head attention                                                          | ID/triplet Loss                                             | 60.27       | 80.01 86.74             |                   |             |              |                   |             |             |             |
| SSAN[65] DSSL[24]     | arXiv21 MM21     | TI TI          | Multi-scale         | ResNet-50 ResNet-50      | Bi-LSTM                   | Region, MASK ID/Compound Ranking (CR) Attention                               | loss 61.37 loss                                             | 59.98       | 80.15 86.73 80.41 87.56 | 54.23             | 72.63       | 79.53        | 43.50 39.05       | 67.80       | 62.60       | 62.60       |
| CFLT[66]              | TIP22            | TI             | global Multi-scale  | ResNet-50                | Bi-GRU BERT               | ID/Alignment/MEC RVCF, Transformer CMPM                                       | loss 60.10                                                  |             | 79.60                   | 86.34             |             |              |                   |             |             |             |
| SRCF-BERT[67]         | ECCV22           | TI Multi-scale |                     | ResNet-50                | BERT                      | ID/Compound Ranking                                                           | 64.04                                                       |             | 82.99 88.81             | 57.18             | 75.01       | 81.49        |                   |             |             |             |
| iVAD[68]              | Neurocomuting22  | TI             | Multi-scale         | VGG-16                   | Bi-LSTM                   | Denoising filter Virtual Attributes decoupling                                | (CR)/SEP loss                                               | 58.35       | 84.84                   |                   |             |              |                   |             |             |             |
| SUM[69]               | KBS22            | TI global      |                     | ResNet-50                | Bi-GRU                    | CMC/clc/ipe Attention ID loss,                                                | loss Ranking loss                                           | 59.22       | 78.22 80.35             |                   |             |              | 41.38             | 67.48       |             |             |
| SAF[70]               | ICASSP22         | TI             | global              | ViT-Base                 | BERT                      | Multi-head attention                                                          |                                                             | 64.13       | 82.62                   | 87.60 88.40       |             |              |                   |             |             |             |
| PMAN[71]              | PRCV22           | TI             | Multi-scale         | ResNet-50                | BERTBase-Uncase           | CMPM loss, KL multi-scale attention CMPM loss                                 | 64.51                                                       |             | 83.14 89.15             |                   |             |              |                   |             |             |             |
| IVT[72]               | ECCV22           | TI Multi-scale | (Text)              | ViT-Base                 | Transformer               | CMPM Loss                                                                     | 65.59                                                       | 83.11       | 89.21                   | 56.04             |             | 80.22        | 46.70             | 70.00       |             |             |
| LBUL[73]              | MM22             | TI global      |                     | ResNet-50                | Bi-GRU                    | Mask+CLIP+NLTK /                                                              | 61.95                                                       |             | 81.16 87.19             |                   | 73.60       |              | 43.35             | 66.85       |             |             |
| TIPCB[5]              | Neurocomuting22  | TI             | global              | ResNet-50                | BERT                      | Ranking loss, ID loss CMPM loss                                               |                                                             | 63.63       | 82.82 89.10             | 54.96             | 74.72       | 81.89        |                   |             |             |             |
| CAIBC[74]             | MM22             | TI             | global              | ResNet-50                | BERT                      | mask Greyscale, color mask                                                    | triplet/ranking/ID loss                                     | 64.43       | 88.37                   | 58.44             | 78.2        | 85.46        | 47.35             | 69.55       | 79.00       | 79.00       |
| LGUR[75]              | MM22             | TI global      |                     | DeiT-Small               | BERT, Bi-LSTM             | mask                                                                          | loss,                                                       | 64.21       | 82.87 81.94             | 87.93             |             |              |                   |             |             |             |
| AXM-Net[76]           | AAAI22           | TI Multi-scale |                     | ResNet-50                | BERT                      | ID/ranking ID/triplet/affinity loss                                           | 64.44                                                       | 80.52       | 86.77                   | 49.37             | 71.69       |              | 53.14             | 74.03       | 81.95       | 81.95       |
| ACSA[77]              | IEEE Trans. MM22 | TI Asymmetric  | Cross-scale         | Swin transformer         | BERT                      | Parts, Self-Attention for Tex Part                                            | ACSA/CMPM-C loss                                            | 63.56       | 87.70                   |                   |             | 79.27        | 48.40             | 71.85       |             |             |
| TFAF[78]              | IEEE SPL22       | TI             | Multi-scale         | Pyramid ViT              | BERT, CNN                 | transformer CMPM/Reconstraction loss                                          | 65.69                                                       | 81.40 84.75 |                         | 89.93             |             |              |                   |             |             |             |
| PDA[2]                | IJCNN22          | TI             | Global              | ResNet-50                | BERT                      | Mask                                                                          | CMPM loss                                                   | 65.26       | 84.58 89.98             | 53.14             |             | 74.03        | 81.95             | 78.70       |             | 86.60       |
| (C2A2)RR[26] SCFC[79] | MM22 ArXiv22     | TI             | Multi-scale         | ResNet-50                | BERT-BaseUncased          | Attribute dictionary Graph CNN, GAN                                           | ID/CMPC loss, KL divergence CMPM loss                       | 67.94       | 86.86 82.76             | 91.87 88.65       |             |              | 54.30             | 70.45       |             |             |
|                       |                  | TI             | Multi-scale         | ResNet-50                | BERT                      |                                                                               | CMPM-CMPC/triplet loss                                      | 64.12       |                         | 60.83             | 76.55       |              | 45.88             |             |             | 72.50       |
| CFine[80]             | arXiv22          | TI, IT         | Multi-scale         | CIIP-ViT                 | BERT                      | CLIP Implicit local alignment                                                 | 63.92                                                       | 69.67       | 85.93 91.15             | 57.73             |             | 75.42        | 82.42 50.55 81.72 | 81.60       | 81.60       | 81.60       |
| ISANet[81]            | arXiv23          |                | Multi-scale         | ResNet-50                | Bi-LSTM                   | integrity                                                                     | ID/ranking/triplet loss ranking/dynamic attribute prompt    |             | 87.69                   |                   |             |              | 75.45             |             |             |             |
| TP-TPS[82]            | arXiv23          | TI TI          | Local               | CLIP-ViT                 | CLIP-Xformer              | CLIP                                                                          | loss 70.16                                                  | 82.15 86.10 | 90.98                   | 54.12             | 75.45       | 82.97        | 54.12             | 82.97       | 82.97       | 82.97       |
| IRRA[83]              | CVPR23           | TI             | Multi-scale         | CLIP-ViT                 | CLIP-Xformer CLIP-Xformer | CLIP, Self/Cross attention ALBEF                                              | ID/SDM/DRR loss Relation/Sensitivity-Aware/contrastive loss | 73.38 76.51 | 89.90 93.71 94.25       | 63.46 65.28       | 80.24 80.40 | 85.82 85.12  | 60.20 66.90       | 81.30 86.50 | 88.20 91.25 | 88.20 91.25 |
| RaSa[84]              | IJCAI23          | TI             | global              | ViT-B-16                 |                           |                                                                               | ID/Contrastive loss                                         | 76.53       | 90.29 90.04             |                   |             | 87.56        | 67.50             |             |             |             |
| APTM[85]              | MM23             | TI             | global              | Swin Transformer         | BERT                      | Attribute, Mask                                                               |                                                             |             |                         | 94.15             | 68.51       | 82.99        |                   |             | 91.45       | 91.45       |

Fig. 8: The figure shows the overall idea and organization regarding the running of the article, which is divided into two parts: survey and foresight. The survey part elucidates the current research on text-based pedestrian re-identification and presents some of the existing challenges. For the open-set retrieval problem, a novel baseline architecture is proposed in the foresight part.

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00096v1-figure-933d22bf-12fa-4fc5-a51f-da1f8602cd2b.jpg)
Fanzhi Jiang Su Yang Mark W. Jones Liumei Zhang
From Attributes to Natural Language: A Survey and Foresight on Text-based Person Re-identification
Text-based person re-identification (Re-ID) is a challenging topic in the
field of complex multimodal analysis, its ultimate aim is to recognize specific
pedestrians by scrutinizing attributes/natural language descriptions. Despite
the wide range of applicable areas such as security surveillance, video
retrieval, person tracking, and social media analytics, there is a notable
absence of comprehensive reviews dedicated to summarizing the text-based person
Re-ID from a technical perspective. To address this gap, we propose to
introduce a taxonomy spanning Evaluation, Strategy, Architecture, and
Optimization dimensions, providing a comprehensive survey of the text-based
person Re-ID task. We start by laying the groundwork for text-based person
Re-ID, elucidating fundamental concepts related to attribute/natural
language-based identification. Then a thorough examination of existing
benchmark datasets and metrics is presented. Subsequently, we further delve
into prevalent feature extraction strategies employed in text-based person
Re-ID research, followed by a concise summary of common network architectures
within the domain. Prevalent loss functions utilized for model optimization and
modality alignment in text-based person Re-ID are also scrutinized. To
conclude, we offer a concise summary of our findings, pinpointing challenges in
text-based person Re-ID. In response to these challenges, we outline potential
avenues for future open-set text-based person Re-ID and present a baseline
architecture for text-based pedestrian image generation-guided
re-identification(TBPGR).