summaries: The paper is tackling the topic of fine-grained representation. In contrast to comparable methods, ARP focus on sub-sampled via strided operation to represent fine-grained information.  ARP is an intuitive strategy to extract critical regions and is more concise compared with multi-stage cascade architecture. This fact facilitates the efficiency. - **Motivation**. The paper argues that    - (a) there are some small but important regions of an interesting object, which are usually ignored by the previous fine-grained methods;   - (b) excessive reduction operations on image resolution fade the discriminative features.     - **Method**. According to the above observations, the paper proposes a pooling algorithm called Adaptive Region Pooling (ARP). This module has two procedures:   - (a) learn to crop image regions;   - (b) downsampling various size regions using bilinear operation.   - **Experiments**. The proposed method is verified in two tasks: fine-grained classification and re-identification.  This paper presents an adaptive region pooling (ARP) method for fine-grained representation learning. ARP crops the features with the estimated scale. The features are further downsampled to a consistent size through bilinear downsampling. Experiments conducted on two tasks validate the effectiveness of the proposed method. This paper proposes an adaptive region pooling (ARP) module that adaptively estimates the scale factors and crops the discriminative regions based on the estimated scale factors. It aims to focus on the most discriminative region and simultaneously contain more fine-grained information. The authors apply the ARP module to fine-grained image recognition and Vehicle Re-Id tasks.
abstractText: Fine-grained recognition aims to discriminate the sub-categories of the images within one general category. It is fundamentally difficult due to the requirement to extract fine-grained features from subtle regions. Nonetheless, a Convolutional Neural Network typically applies strided operations to downsample the representation, which would excessively spoil the feature resolution and lead to a significant loss of fine-grained information. In this paper, we propose Adaptive Region Pooling (ARP): a novel downsampling algorithm that makes the network only focus on a smaller but more critical region, and simultaneously increase the resolution of sub-sampled feature. ARP owns a trade-off mechanism that allows users to actively balance the scale of receptive field and the granularity of feature. Also, without any learning-based parameters, ARP provides the network a stabler training process and an earlier convergence. Extensive experiments qualitatively and quantitatively validate the effectiveness and efficiency of the proposed pooling operation and show superior performance against the state-of-the-arts in both the tasks of image classification and image retrieval.
references.authors: Binghui Chen Weihong Deng
references.title: Hybrid-attention based decoupled metric learning for zero-shot image retrieval
references.venue: In IEEE Conference on Computer Vision and Pattern Recognition,
references.authors: Long Chen Hanwang Zhang Jun Xiao Liqiang Nie Jian Shao Wei Liu Tat-Seng Chua
references.title: Scacnn: Spatial and channel-wise attention in convolutional networks for image captioning
references.venue: In IEEE Conference on Computer Vision and Pattern Recognition,
references.authors: Tsai-Shien Chen Man-Yu Lee Chih-Ting Liu Shao-Yi Chien
references.title: Viewpoint-aware channel-wise attentive network for vehicle re-identification
references.venue: In IEEE Conference on Computer Vision and Pattern Recognition Workshops,
references.authors: Tsai-Shien Chen Chih-Ting Liu Chih-Wei Wu Shao-Yi Chien
references.title: Orientation-aware vehicle re-identification with semantics-guided part attention network
references.venue: In European Conference on Computer Vision, 2020b
references.authors: Yao Ding Yanzhao Zhou Yi Zhu Qixiang Ye Jianbin Jiao
references.title: Selective sparse sampling for fine-grained image recognition
references.venue: In IEEE International Conference on Computer Vision,
references.authors: Jianlong Fu Heliang Zheng Tao Mei
references.title: Look closer to see better: Recurrent attention convolutional neural network for fine-grained image recognition
references.venue: In IEEE Conference on Computer Vision and Pattern Recognition,
references.authors: Bing He Jia Li Yifan Zhao Yonghong Tian
references.title: Part-regularized near-duplicate vehicle reidentification
references.venue: In IEEE Conference on Computer Vision and Pattern Recognition,
references.authors: Kaiming He Xiangyu Zhang Shaoqing Ren Jian Sun
references.title: Deep residual learning for image recognition
references.venue: In IEEE Conference on Computer Vision and Pattern Recognition,
references.authors: Alexander Hermans Lucas Beyer Bastian Leibe
references.title: In defense of the triplet loss for person reidentification
references.venue: arXiv preprint arXiv:1703.07737,
references.authors: Shaoli Huang Zhe Xu Dacheng Tao Ya Zhang
references.title: Part-stacked cnn for fine-grained visual categorization
references.venue: In IEEE Conference on Computer Vision and Pattern Recognition,
references.authors: Max Jaderberg Karen Simonyan Andrew Zisserman
references.title: Spatial transformer networks
references.venue: In Advances in Neural Information Processing Systems,
references.authors: Ruyi Ji Longyin Wen Libo Zhang Dawei Du Yanjun Wu Chen Zhao Xianglong Liu Feiyue Huang
references.title: Attention convolutional binary neural tree for fine-grained visual categorization
references.venue: In IEEE Conference on Computer Vision and Pattern Recognition,
references.authors: Mahdi M Kalayeh Emrah Basaran Muhittin Gökmen Mustafa E Kamasak Mubarak Shah
references.title: Human semantic parsing for person re-identification
references.venue: In IEEE Conference on Computer Vision and Pattern Recognition,
references.authors: Pirazh Khorramshahi Amit Kumar Neehar Peri Sai Saketh Rambhatla Jun-Cheng Chen Rama Chellappa
references.title: A dual-path model with adaptive attention for vehicle re-identification
references.venue: In IEEE International Conference on Computer Vision,
references.authors: Pirazh Khorramshahi Neehar Peri Jun-cheng Chen Rama Chellappa
references.title: The devil is in the details: Self-supervised attention for vehicle re-identification
references.venue: arXiv preprint arXiv:2004.06271,
references.authors: Jin-Hwa Kim Jaehyun Jun Byoung-Tak Zhang
references.title: Bilinear attention networks
references.venue: In Advances in Neural Information Processing Systems,
references.authors: Jonathan Krause Michael Stark Jia Deng Li Fei-Fei
references.title: 3d object representations for fine-grained categorization
references.venue: In IEEE International Conference on Computer Vision Workshops,
references.authors: Yann LeCun Léon Bottou Yoshua Bengio Patrick Haffner
references.title: Gradient-based learning applied to document recognition
references.venue: Proceedings of the IEEE,
references.authors: Di Lin Xiaoyong Shen Cewu Lu Jiaya Jia
references.title: Deep lac: Deep localization, alignment and classification for fine-grained recognition
references.venue: In IEEE Conference on Computer Vision and Pattern Recognition,
references.authors: Tsung-Yu Lin Aruni RoyChowdhury Subhransu Maji
references.title: Bilinear cnn models for fine-grained visual recognition
references.venue: In IEEE International Conference on Computer Vision,
references.authors: Chih-Ting Liu Man-Yu Lee Tsai-Shien Chen Shao-Yi Chien
references.title: Hard samples rectification for unsupervised cross-domain person re-identification
references.venue: arXiv preprint arXiv:2106.07204,
references.authors: Hongye Liu Yonghong Tian Yaowei Wang Lu Pang Tiejun Huang
references.title: Deep relative distance learning: Tell the difference between similar vehicles
references.venue: In IEEE Conference on Computer Vision and Pattern Recognition,
references.authors: Xiaobin Liu Shiliang Zhang Qingming Huang Wen Gao
references.title: Ram: a region-aware deep model for vehicle re-identification
references.venue: In IEEE International Conference on Multimedia and Expo,
references.authors: Xinchen Liu Wu Liu Huadong Ma Huiyuan Fu
references.title: Large-scale vehicle re-identification in urban surveillance videos
references.venue: In IEEE International Conference on Multimedia and Expo,
references.authors: Xinchen Liu Wu Liu Tao Mei Huadong Ma
references.title: A deep learning-based approach to progressive vehicle re-identification for urban surveillance
references.venue: In European Conference on Computer Vision, 2016c
references.authors: Yihang Lou Yan Bai Jun Liu Shiqi Wang Ling-Yu Duan
references.title: Embedding adversarial learning for vehicle re-identification
references.venue: IEEE Transactions on Image Processing,
references.authors: Hao Luo Youzhi Gu Xingyu Liao Shenqi Lai Wei Jiang
references.title: Bag of tricks and a strong baseline for deep person re-identification
references.venue: In IEEE Conference on Computer Vision and Pattern Recognition Workshops,
references.authors: Wei Luo Xitong Yang Xianjie Mo Yuheng Lu Larry S Davis Jun Li Jian Yang Ser-Nam Lim
references.title: Cross-x learning for fine-grained visual categorization
references.venue: In IEEE Conference on Computer Vision and Pattern Recognition,
references.authors: Subhransu Maji Esa Rahtu Juho Kannala Matthew Blaschko Andrea Vedaldi
references.title: Fine-grained visual classification of aircraft
references.venue: arXiv preprint arXiv:1306.5151,
references.authors: Dechao Meng Liang Li Xuejing Liu Yadong Li Shijie Yang Zheng-Jun Zha Xingyu Gao Shuhui Wang Qingming Huang
references.title: Parsing-based view-aware embedding network for vehicle reidentification
references.venue: In IEEE Conference on Computer Vision and Pattern Recognition,
references.authors: Yuval Netzer Tao Wang Adam Coates Alessandro Bissacco Bo Wu Andrew Y Ng
references.title: Reading digits in natural images with unsupervised feature learning
references.authors: Ergys Ristani Francesco Solera Roger Zou Rita Cucchiara Carlo Tomasi
references.title: Performance measures and a data set for multi-target, multi-camera tracking
references.venue: In ECCV Workshop on Benchmarking Multi-Target Tracking,
references.authors: Karen Simonyan Andrew Zisserman
references.title: Very deep convolutional networks for large-scale image recognition
references.venue: arXiv preprint arXiv:1409.1556,
references.authors: Yifan Sun Liang Zheng Yi Yang Qi Tian Shengjin Wang
references.title: Beyond part models: Person retrieval with refined part pooling (and a strong convolutional baseline)
references.venue: In European Conference on Computer Vision, 2018
references.authors: Dmitry Ulyanov Andrea Vedaldi Victor Lempitsky
references.title: Instance normalization: The missing ingredient for fast stylization
references.venue: arXiv preprint arXiv:1607.08022,
references.authors: Catherine Wah Steve Branson Peter Welinder Pietro Perona Serge Belongie
references.title: The caltech-ucsd birds-200-2011 dataset
references.authors: Yaming Wang Vlad I Morariu Larry S Davis
references.title: Learning a discriminative filter bank within a cnn for fine-grained recognition
references.venue: In IEEE Conference on Computer Vision and Pattern Recognition,
references.authors: Zhongdao Wang Luming Tang Xihui Liu Zhuliang Yao Shuai Yi Jing Shao Junjie Yan Shengjin Wang Hongsheng Li Xiaogang Wang
references.title: Orientation invariant feature embedding and spatial temporal regularization for vehicle re-identification
references.venue: In IEEE International Conference on Computer Vision,
references.authors: Peter Welinder Steve Branson Takeshi Mita Catherine Wah Florian Schroff Serge Belongie Pietro Perona
references.title: Caltech-ucsd birds
references.authors: Saining Xie Ross Girshick Piotr Dollár Zhuowen Tu Kaiming He
references.title: Aggregated residual transformations for deep neural networks
references.venue: In IEEE Conference on Computer Vision and Pattern Recognition,
references.authors: Hantao Yao Shiliang Zhang Richang Hong Yongdong Zhang Changsheng Xu Qi Tian
references.title: Deep representation learning with part loss for person re-identification
references.venue: IEEE Transactions on Image Processing,
references.authors: Yue Yao Liang Zheng Xiaodong Yang Milind Naphade Tom Gedeon
references.title: Simulating content consistent vehicle datasets with attribute descent
references.venue: In European Conference on Computer Vision,
references.authors: Chaojian Yu Xinyi Zhao Qi Zheng Peng Zhang Xinge You
references.title: Hierarchical bilinear pooling for fine-grained visual recognition
references.venue: In European Conference on Computer Vision,
references.authors: Lianbo Zhang Shaoli Huang Wei Liu Dacheng Tao
references.title: Learning a mixture of granularity-specific experts for fine-grained categorization
references.venue: In IEEE International Conference on Computer Vision, 2019
references.authors: Ning Zhang Ryan Farrell Forrest Iandola Trevor Darrell
references.title: Deformable part descriptors for fine-grained recognition and attribute prediction
references.venue: In IEEE International Conference on Computer Vision,
references.authors: Ning Zhang Jeff Donahue Ross Girshick Trevor Darrell
references.title: Part-based r-cnns for fine-grained category detection
references.venue: In European Conference on Computer Vision,
references.authors: Heliang Zheng Jianlong Fu Tao Mei Jiebo Luo
references.title: Learning multi-attention convolutional neural network for fine-grained image recognition
references.venue: In IEEE International Conference on Computer Vision,
references.authors: Heliang Zheng Jianlong Fu Zheng-Jun Zha Jiebo Luo
references.title: Looking for the devil in the details: Learning trilinear attention sampling network for fine-grained image recognition
references.venue: In IEEE Conference on Computer Vision and Pattern Recognition,
references.authors: Liang Zheng Liyue Shen Lu Tian Shengjin Wang Jingdong Wang Qi Tian
references.title: Scalable person re-identification: A benchmark
references.venue: In IEEE International Conference on Computer Vision,
references.authors: Zhedong Zheng Liang Zheng Yi Yang
references.title: Unlabeled samples generated by gan improve the person re-identification baseline in vitro
references.venue: In IEEE International Conference on Computer Vision,
references.authors: Yi Zhou Ling Shao
references.title: Viewpoint-aware attentive multi-view inference for vehicle reidentification
references.venue: In IEEE Conference on Computer Vision and Pattern Recognition,
sections.heading: 1 INTRODUCTION
sections.text: The main goal of fine-grained recognition is to discriminate the sub-categories of the images that belong to a same general class. Its practical applications include fine-grained image classification (Welinder et al., 2010; Krause et al., 2013; Maji et al., 2013), re-identification (Zheng et al., 2017b; Liu et al., 2016b;a), etc. These tasks are fundamentally challenging due to considerable inter-class similarity. Consequently, an ideal feature extractor should be capable of extracting discriminative fine-grained features from subtle image regions, such as the beak to classify the bird species or the stickers on the windshield to identify the vehicle.
  While the general design of Convolutional Neural Network (CNN) (Simonyan & Zisserman, 2014; He et al., 2016) applies either strided convolution or strided pooling as downsampling operations to reduce the computational requirement of the network and also increase the receptive field of subsequent convolutions, these operations, however, have been substantiated to be harmful to fine-grained recognition (Sun et al., 2018; Luo et al., 2019a). Its disadvantage is two-fold: 1) false assumption on spatial equality which would overemphasize the background region while neglecting the smaller but more discriminative regions and 2) excessive reduction of resolution which would spoil the granularity of feature. To cope with the problem, existing approaches (Zhang et al., 2014; Fu et al., 2017; Chen & Deng, 2019; Zhang et al., 2019) introduce cropping operation that guides the network to only focus on the most critical region while preserving more details in the sampled part. However, to maintain computational complexity, full conservation of feature resolution would instead restrict a limited scale, or area, of the cropped region. Comprehensively, as shown in Figure 1(a)(c), the problems of two commonly used downsamplings can be summarized into a trade-off problem between the scale of receptive field and the granularity of representation.
  To tackle the problem, in this work, we introduce a novel pooling algorithm, named Adaptive Region Pooling (ARP), to bridge strided and cropping operations. It consists of two steps. First, ARP automatically crops the feature from the most critical region with a well-estimated and adaptive cropping scale. Second, after being cropped into different sizes, the feature is further downsampled to a consistent size through bilinear downsampling. As illustrated in Figure 1(b), with the meticulous evaluation of cropping scale, ARP can better handle the trade-off issue and leverage the
  beneficial properties of both strided and cropping operations. Besides, ARP also features other three advantages. First, it is incorporated with a controllable mechanism that supports users to actively reach a more satisfying balance in the trade-off problem, as in two cases in Figure 1(b). Second, compared to the previous work (Lin et al., 2015a; Huang et al., 2016; Kim et al., 2018; Zheng et al., 2019) which implement the cropping operations by an additional neural network, ARP utilizes a more efficient yet effective operation without any learning-based parameters. Hence, it makes the network achieve lower computational overhead, experience a stabler training process, and converge more quickly. Finally, ARP can be simply plugged into any CNN layer and the network can be optimized in a one-stage end-to-end manner, which is completely identical to the training of original CNN backbone.
  At last, we demonstrate that ARP could facilitate fine-grained recognition through our Multiple Scale and Granularity Network (MSGN). In MSGN, ARP serves as a downsampling alternative to extract the feature of finer details complementary to the coarse feature provided by the typical downsampling method. The contributions of this paper can be summarized as follows:
  • We introduce Adaptive Region Pooling (ARP): a novel downsampling that smoothly bridges strided pooling and cropping operations. The sub-sampled representation focuses on the most discriminative region and simultaneously contains more fine-grained information. • To our knowledge, ARP is the first pooling algorithm with an adjustable trade-off mechanism that allows users to apply the human domain knowledge of the target task to strike a more desirable balance between the scale of receptive field and the granularity of downsampled feature. • With the usage of ARP, MSGN outperforms the existing frameworks in terms of both effectiveness and efficiency on multiple benchmarks for both the tasks of image classification and retrieval.
sections.heading: 2 RELATED WORK
sections.text: Fine-grained recognition focuses on learning discriminative representations to classify or identify the images within a same general category. There are two common applications: re-identification and fine-grained image classification.
  Re-identification (re-ID) studies the problem of identifying the individuals of persons (Zheng et al., 2015; 2017b; Ristani et al., 2016) or vehicles (Liu et al., 2016b;c;a) in different camera views. To distinguish two similar targets with only slight differences, most of the existing methods (Yao et al., 2019; Kalayeh et al., 2018; Liu et al., 2021; Wang et al., 2017; Zhou & Shao, 2018; Chen et al., 2020b) adopt spatial attentive mechanism to emphasize the features extracted from critical parts, such as the head for person re-ID or the tires for vehicle re-ID. While previous work get significant improvement, they generally suffer from massive loss of fine-grained details when CNN backbone downsamples the feature through strided operations. Sun et al. (2018) and Luo et al. (2019a) both
  find a huge performance gain in the task of person re-ID, numerically 1.4% for mAP on Market1501 (Zheng et al., 2015), by simply removing the pooling operation in the last CNN layer.
  Fine-grained image classification (FGIC) aims to classify the sub-classes of images, such as the species of birds (Welinder et al., 2010) or the models of vehicles (Krause et al., 2013). Compared to re-ID, the datasets of FGIC have a severe problem of image misalignment which impels lots of work to apply another sampling operation, cropping, to align the image or representation and also guide the network to focus on the sampled region. Several prior work (Welinder et al., 2010; Zhang et al., 2013; 2014; Lin et al., 2015a; Huang et al., 2016; Kim et al., 2018; He et al., 2019) leverage extra object detection datasets or human-labeled bounding box annotations to learn the localization of discriminative regions. However, such auxiliary labels are hard to acquire in the real-world and large-scale scenarios. Recently, there have been emerging work with a more general setting by proposing unsupervised approaches. Jaderberg et al. (2015) introduce a differentiable STN that can learn to well align the images or representations by themselves. While it can perform well on the digit datasets (LeCun et al., 1998; Netzer et al., 2011), lack of explicit guidance leads to the unstable training process and being hard to converge on more sophisticated tasks (see Section 4.3 for more details and discussions). As the following work, Fu et al. (2017) propose RACNN which is a cascade coarse-to-fine CNN model. The similar architecture is also adopted by Chen & Deng (2019) and Zhang et al. (2019). Although they show the great capability of cropping the discriminative region, the cascade architecture requires multi-stage training for better optimization and struggles with the requirement of heavy computation. In this work, we also introduce a sampling operation with an unsupervised approach; however, Adaptive Region Pooling (ARP) contains neither additional neural network nor extra learning-based parameters and makes the network using ARP more computationefficient and able to be optimized in a simple one-stage end-to-end manner.
sections.heading: 3 PROPOSED METHODOLOGY
sections.text: In this section, we introduce the proposed pooling algorithm in three parts. First, in Section 3.1, Region Pooling (RP) considers a simplified task which aims to crop the hidden representation from the most critical region with a fixed cropping scale. Then, in Section 3.2, Adaptive Region Pooling (ARP) estimates the adaptive cropping scale for each representation to better capture the interesting object. Finally, we introduce Multiple Scale and Granularity Network (MSGN) as an example of network architecture with the usage of ARP in Section 3.3.
sections.heading: 3.1 REGION POOLING
sections.text: Given a CNN feature and a fixed downsampling rate, the main goal of Region Pooling (RP) is to crop the feature from the most critical region into a consistent scale. An intuitive approach is to use an extra network to predict the cropping center which nevertheless, often suffers from the requirement of expensive bounding box annotations or complicated multi-stage learning strategy to train the network. In contrast, RP uses a more efficient yet effective operation with two steps.
  In the first step, as shown in Figure 2 (a), RP would evaluate the significance score for each location and generate the keypoint confidence map C based on the C-dim feature vectors, where C denotes the number of channels. Here, we define “significance” in an engineering sense as follows: a location, whose feature vector includes more highly-activated neurons, is more significant. The physical meaning is that a channel-wise feature map is actually a detector of a certain semantic object (Chen et al., 2017; 2020a), such as a bird’s beak or wings. For a well-trained CNN, most of the channels would detect the objects which are more beneficial to the target task; thus, the locations activated by more channels typically cover critical semantic objects and are more significant.
  Based on the definition, we then give a formal formulation as follows. For the input feature F ∈ RC×H×W with width W , height H and C channels and its element (or activated value) denoted as x(i,j,k) at ith channel, jth row and kth column, in order to balance the influence of each channel in the subsequent operations, we first standardize each channel-wise feature map as:
  x̂(i,j,k) = x(i,j,k) − µ(i)
  σ(i) , where µ(i) =
  H∑ j W∑ k x(i,j,k)
  H ·W , σ2(i) =
  H∑ j W∑ k (x(i,j,k) − µ(i))2
  H ·W , (1)
  and x̂ represents the element of the standardized feature. Such standardization can be simply implemented by Instance Normalization (Ulyanov et al., 2016). Afterwards, we evaluate the significance score of each location based on the average of their feature vectors with the form as:
  s(j,k) =
  C∑ i=1 x̂(i,j,k)
  C (2)
  where s(j,k) represents the significance score of the location (j, k). Complied with our definition, Equation 2 assigns larger s for the locations that are highly activated in more channels. Next, we estimate the confidence of a location to be a “keypoint”, or a critical location, by Sigmoid function σ(·). Specifically, we generate a keypoint confidence map C ∈ RH×W by:
  C = [ c(1,1), ..., c(H,W ) ] = [ σ(s(1,1)), ..., σ(s(H,W )) ] (3)
  where c(j,k) denotes the keypoint confidence at the location (j, k). We visualize the generated keypoint confidence maps C in Figure 5 and 6. If we project them to the original images, the locations with larger keypoint confidences effectively represent the more discriminative regions, such as the beak and colorful feather of a bird or the bumper and lamp of a vehicle.
  In the second step, as in Figure 2(c), given a fixed downsampling rate DSR = [DSRH , DSRW ], RP would crop the most critical region with height (H · DSRH ) and width (W · DSRW ). To this end, we apply a convolution operation on the keypoint confidence map C with a kernel K ∈ R(H·DSRH)×(W ·DSRW ). The region with the largest weighted summation of keypoint confidences would be selected as the cropped region, which can be written as:
  (j∗, k∗) = arg max (j,k) (K C(j,k)) (4)
  where represents inner product, C(j,k) is the cropped keypoint confidence map centered at the location (j, k), and (j∗, k∗) indicates the center location of the selected cropped region.
  Discussion of kernel K. Empirically, we find that using Gaussian filter as kernel K yields the best performance. A potential explanation is that it tends to select the region with more keypoints locating near the center of cropped region and hence, also solves the problem of feature misalignment.
sections.heading: 3.2 ADAPTIVE REGION POOLING
sections.text: Although Region Pooling demonstrates the great ability to sample the feature from the most critical region, nonetheless, there is no single downsampling rate suitable for all features. Specifically, the features with highly separate keypoints should have a larger downsampling rate to cover a wider interesting region while in contrast, the ones with more concentrated keypoints should use a smaller rate. To meet this requirement, we further introduce Adaptive Region Pooling (ARP) that automatically estimates a better adaptive downsampling rate based on the distribution of keypoints, or more precisely: the standard deviation (abbreviated as std) of keypoints distribution.
  As depicted in Figure 2(b), to evaluate the adaptive downsampling rate of height DSRadpH or width DSRadpW , ARP respectively refers the vertical or horizontal normalized distribution of keypoints:
  Dv = [ W∑ k=1 c(1,k) Σ C , ..., W∑ k=1 c(H,k) Σ C ] ∈ RH , Dh =  H∑ j=1 c(j,1) Σ C , ..., H∑ j=1 c(j,W ) Σ C  ∈ RW (5) where ΣC is the summation of keypoint confidence map.
  Since the derivations of DSRadpH and DSR adp W take similar forms, we use the former as an example in the following. Here, we first consider a simplified condition that each image only contains one consecutive interesting object and the keypoints uniformly distribute in the target object. We will explore more circumstances in the following discussion and Appendix A. Under this assumption, the keypoints distribution can be regarded as a discrete uniform distribution:
  Dv(y) =  1b− a , a < y 6 b0 , otherwise , ∀ 0 ≤ a < b ≤ H, (6) with its std:
  σ(Dv) = √
  (b− a)2 − 1 12 ≈ b− a√ 12 . (7)
  Note that Dv has the keypoints all lying within the interval (a, b]. Therefore, if the cropped region is expected to cover all keypoints while ignoring the background, the ideal length of cropped region should be (b− a) and hence, the expected downsampling rate should be (b− a)/H . Combing with Equation 7, we can rewrite the adaptive downsampling rate as:
  DSRadpH = b− a H =
  √ 12
  H · σ(Dv). (8)
  Finally, with Equation 8, we can use the std of keypoints distribution to well estimate the adaptive downsampling rate that could exactly include all keypoints in the cropped region.
  Discussion of different distributions D. Intuitively, in Equation 8, the adaptive downsampling rate is proportional to the std of keypoints distribution. Therefore, for more sparse or dense keypoints distribution (e.g., M-shaped or bell-shaped distribution), the cropped region could automatically be wider or narrower to better capture the interesting object. In Appendix A, we also empirically test the stability and rationality of Equation 8 on various randomly simulated distributions and show that Equation 8 can consistently evaluate an intelligible adaptive downsampling rate that can exactly cover most of the keypoints.
  Subsequently, as in Figure 2, ARP first crops the feature F from the most critical region based on Equation 4 and adaptive downsampling rate as Fcrop ∈ RC×(H·DSR adp H )×(W ·DSR adp W ) and then, subsamples it to a consistent size as F′ ∈ RC×(H·DSRH)×(W ·DSRW ) through bilinear downsampling. Such a “cropping and downsampling” process can be viewed as the trade-off between the scale of receptive field and the resolution of feature. Specifically, the feature with a larger cropped region would encounter a huge reduction of the feature granularity in the following downsampling which makes ARP act as strided operation. In contrast, a smaller cropped region would preserve more finegrained details in the downsampled feature; in such a case, ARP intrinsically behaves like cropping function. To allow users to strike a better balance in the trade-off problem, we further incorporate ARP with a controllable mechanism by revising Equation 8 as:
  DSRadpH = min
  ( max ( k · √ 12
  H · σ(Dv), DSRH
  ) , 1 ) . (9)
  k is the trade-off coefficient and min-max operation ensures a reasonable downsampling rate.
  Discussion of trade-off coefficient k. k is a meaningful coefficient. When it is set to 1, the cropped region would nearly cover the whole target object, such as an entire bird or vehicle. And if a smaller k is set, the length of cropped region would be proportionally decreased. Through k, users can actively reach a satisfying sampling scale according to the human domain knowledge of the target task. Take the below case of Figure 1(b) as an example, users can set a smaller k to extract the feature only from the front face of vehicle, which usually contains many identity-relative features, rather than the whole vehicle. Hence, the sub-sampled feature would keep richer fine-grained details.
sections.heading: 3.3 MULTIPLE SCALE AND GRANULARITY NETWORK
sections.text: We introduce Multiple Scale and Granularity Network (MSGN) as an example of the network architecture that uses ARP as the downsampling operations. As in Figure 3, MSGN has one low-level feature extractor followed by multiple parallel high-level extractors. Considering that the hidden representations before Conv3 merely contain the information of low-level patterns which are unreliable to determine the critical region, we only adopt ARP with k = 1 in Conv3 to remove some background regions in the early stage. As for the following high-level extractors, including one global and several local view branches, we respectively apply strided operation and ARP with smaller k in Conv4 to downsample the feature with different scales and granularities. We also follow the strategy from Sun et al. (2018) and Luo et al. (2019a) to remove pooling operations in Conv5 of local view branches. Finally, the features extracted from the multiple branches would be concatenated and fed into a fully connected layer to comprehensively generate the final representative embedding by referring to the features from both global and local views. We will first validate MSGN with one local view branch in Section 4 and further, discuss the network with more branches in Appendix B.
sections.heading: 4 EXPERIMENTS
sections.text: In the following, we evaluate ARP through MSGN in both tasks of fine-grained image classification (FGIC) and vehicle re-identification (re-ID) for image retrieval.
sections.heading: 4.1 IMPLEMENTATION DETAILS
sections.text: We implement MSGN with three backbones: ResNet-50, ResNet-101 and ResNeXt-101 32x8d (He et al., 2016; Xie et al., 2017). For a fair comparison, we set DSR of ARP as [0.5, 0.5] to share the same height and width of the downsampled feature as from the pooling with stride = 2. Following the same protocol by Zhang et al. (2019) and Ji et al. (2020), we augment input images by resizing to 512× 512 then randomly cropping to 448× 448 with random horizontal flipping and use crossentropy to supervise the training. For vehicle re-ID, as in standard literature (Luo et al., 2019a; Chen et al., 2020b), we directly resize images to 224× 224 and use the combinations of batch-hard triplet loss (Hermans et al., 2017) and cross-entropy loss for the supervision. The training lasts for 100 epochs. More training and testing details can be found in Appendix C.
sections.heading: 4.2 COMPARISON WITH THE STATE-OF-THE-ARTS
sections.text: Fine-grained image classification. We first evaluate MSGN on FGIC in terms of both effectiveness and efficiency and report the results in Table 1. MSGN achieves the new state-of-the-art performance: 90.2% on CUB-200-2011, 95.6% on Stanford Cars, and 94.4% on FGVC-Aircraft with ResNeXt-101. With ResNet-50 or ResNet-101, MSGN also shows a consistent improvement over the previous counterparts with the same CNN backbone. In addition to effectiveness, our model also achieves a significant enhancement in efficiency. While previous work apply the additional network
  to generate the sub-sampling parameters (Lin et al., 2015a; Zheng et al., 2019; Ding et al., 2019) or implement the cascade coarse-to-fine architecture (Fu et al., 2017; Chen & Deng, 2019; Zhang et al., 2019), ARP only utilizes efficient operations for sampling. Therefore, the computational cost of MSGN is approximately half of the previous models, specifically 44.5% lower than CrossX (Luo et al., 2019b) on ResNet-50 and 52.7% lower than MGE (Zhang et al., 2019) on ResNet-101.
  Vehicle re-identification. We also evaluate the representation learning of MSGN on vehicle reID and report the results in Table 2. Prior approaches (Wang et al., 2017; Zhou & Shao, 2018; Meng et al., 2020; Chen et al., 2020b) mostly focus on spatial attentive mechanism but ignoring the loss of fine-grained information when the representation are sub-sampled through strided operation. In contrast, MSGN effectively solves the problem by using ARP as the downsampling operation. Hence, MSGN outperforms the previous frameworks on both VeRi-776 and VehicleID, especially on VeRi-776 by a notable margin of 2.2% on mAP compared to SAVER (Khorramshahi et al., 2020).
sections.heading: 4.3 COMPARISON WITH SPATIAL TRANSFORMER NETWORK
sections.text: Spatial Transformer (ST) (Jaderberg et al., 2015) module is a differentiable sampling operation that can automatically learn to perform an affine transformation on an image or a hidden representation. In light of the similar functionality between ST module and ARP, we additionally compare these two poolings. In the experiment, we adopt ST module or ARP as the downsampling operation in Conv4 of local view branch in MSGN and plot the learning curves and the learned sampling outcomes in Figure 4. We then discuss the comparison under two aspects: stability and intervenability.
  First, as shown in Figure 4(a), the network using ST module comparably experiences a slower learning progress, reaches later convergence, and eventually performs worse at the end of training. We can find a possible explanation from Figure 4(b); since ST module relies on learning-based parameters for sampling, in the first ten epochs, it performs unstably and generates unaccountable downsampling results which instead disturb the training. In contrast, ARP is able to correctly sample from the foreground object (red box in Figure 4(b)) even in the first epoch. Following the training, with more channels detecting the beneficial vehicle parts, the sampled region progressively covers the entire front face of vehicle which contains more identity-relative features. It is also worth noting that due to the instability of ST module, in the paper (Jaderberg et al., 2015), ST module is restricted to perform the 2 DoF transformation (i.e., only translation) in the harder task, like FGIC.
  Next, at the end of the training, ST module fails to learn an effective transformation which is supposed to concentrate on the more critical region. It leads to the subsequent discussion: for learningbased cropping operations, users can only passively accept the training results and lack interpretable parameters to adjust the sampling operation to better fit the target task. Conversely, owing to a meaningful trade-off mechanism, ARP allows users to actively use the prior knowledge to intervene in the representation learning and reach the more desirable sub-sampling characteristic.
sections.heading: 4.4 ABLATION STUDIES
sections.text: We evaluate the effectiveness of each component in the proposed framework and report the results in Table 3. The baseline model simply uses the original ResNet-50 as the extractor. We then replace the downsampling in Conv4 with ARP of k = 0.9. Leveraging the attention on more discriminative regions, the model achieves the improvement of 0.6% on CUB-200-2011 and 0.5% on VeRi-776. Next, as listed in the third to fifth rows in Table 3, we apply another high-level extractor with three different downsampling operations in Conv4, including strided convolution, RP with DSR = [0.5, 0.5], and ARP with k = 0.7. It shows that, with a parallel branch, the performances are all boosted. However, without using RP and ARP, the network comparably performs worse due to the lack of fine-grained information. Furthermore, the network with ARP yields the best performance, which describes that ARP can adaptively estimate the better downsampling rate for each representation and properly cover the critical region. Finally, with all proposed components, MSGN with ARP outperforms the baseline model by 3.5% on CUB-200-2011 and 2.3% on VeRi-776.
sections.heading: 4.5 VISUALIZATION
sections.text: Qualitative results of Adaptive Region Pooling. To understand the practical effect of ARP, we plot the keypoint confidence maps C and the cropped regions in Figure 5 and 6 (especially for the final setting of k = 0.7). Note that the demonstrated images are in various manners, such as different actions of birds (e.g., standing and flying) and different viewpoints or types of vehicles (e.g., truck, bus, SUV, and sedan) to validate the robustness of ARP. First, as in Figures 5(b) and 6, the keypoint confidence maps C from Conv4 are able to recognize the critical locations in images, such as the bird’s beak and colorful feathers or the vehicle’s bumper and decoration. Along with the proper adaptive downsampling rate, the sampled regions can cover the most critical and consistent part in different images. Also, as in Figure 5(a), ARP in Conv3 can eliminate some negligible background regions in the early stage while enriching the feature resolution in the foreground object.
  Selection of trade-off coefficient k. We visualize the cropped regions and list the validation results with different k in Figure 6. As in the second to fourth columns, users can set k = 0.5 or 0.7 or 0.9 if the sampled region is respectively expected to cover merely a part of front face, or the whole front face with the windshield, or nearly the entire vehicle. Commonly, the front face of vehicle includes several important but fine identity-relative features which are beneficial to vehicle re-ID. Hence, with the aid of domain knowledge, the final setting of k = 0.7 yields the best performance.
sections.heading: 5 CONCLUSION
sections.text: In this paper, we propose a novel downsampling operation, Adaptive Region Pooling (ARP), which automatically samples the feature from the most critical region and in the same time, increases the feature granularity. Besides, ARP owns an adjustable trade-off mechanism which supports users to actively balance the scale of receptive field and the resolution of sub-sampled feature. Without any learning-based parameters, ARP can be simply integrated into a CNN backbone and the network can be stably end-to-end optimized. The experiments show the effectiveness and efficiency of the proposed model in both the tasks of image classification and image retrieval.
sections.heading: A ROBUSTNESS TESTING OF ADAPTIVE DOWNSAMPLING RATE
sections.text: In Equation 8, we show that the adaptive downsampling rate of height or width can be respectively derived by the standard deviation (abbreviated as std) of 1-D vertical or horizontal keypoints distribution. In this section, we further evaluate the stability and rationality of Equation 8 on the various simulated keypoints distributions. We comply with the following rules to simulate the distributions: 1) most of the keypoints locate in one consecutive interval; 2) the interval can have any length and can be anywhere in the entire distribution; 3) the distribution within the interval is random. These rules reflect the fact that the images for fine-grained recognition typically contain only one target object (e.g., a bird or a vehicle) with various scales and locations, and the critical features (e.g., a bird’s beak or a vehicle’s tires) are randomly distributed in the target object. Following these principles, we simulate the distributions by first assigning a random significance score at each location and filtering by a 1-D Gaussian kernel with a random mean and std. We plot the distributions and also show the cropped lengths and regions in Figure 7.
  We can observe that for each keypoints distribution with different sparsity or at a different location, Equation 8 can consistently evaluate an appropriate adaptive downsampling rate that can cover most of the keypoints while removing the negligible background region with a relatively few keypoints. Subsequently, through the trade-off mechanism, users can actively adjust the scale of cropped region to better capture the interesting parts in the target object.
sections.heading: B MULTIPLE SCALE AND GRANULARITY NETWORK WITH MORE LOCAL VIEW BRANCHES
sections.text: In Section 4, we have shown the effectiveness of MSGN with one local view branch (LVB) and will further discuss the behavior of MSGN with more (i.e. 2 and 4) LVBs in the following. To avoid excessive overlaps among the sampled regions from different LVBs, we set a smaller k = 0.5 for Adaptive Region Pooling (ARP) in Conv4 to reduce the scale of cropped region. We evaluate the models on two datasets: CUB-200-2011 (Wah et al., 2011) and VeRi-776 (Liu et al., 2016b;c), and respectively report the visualization results and the validation performance in Figure 8 and Table 4.
  We observe that, for MSGN with two LVBs, two branches can learn to focus on the scattered parts in the target object, such as the head and torso of a bird, and can jointly cover larger critical regions
  in contrast to the one-branch model (by the comparison with the visualization in Figure 5). Also, due to smaller k, the model can leverage the higher resolution of sub-sampled feature and the richer details in the cropped region. Therefore, as listed in Table 4, we can get a performance gain when we increase the number of LVB from one to two. However, there is no more improvement when we further apply four LVBs. A possible explanation is that the sampled regions from the two-branches model have already covered most of the parts in the interesting object; therefore, the extra LVBs would merely focus on the overlapping regions and provide redundant information.
sections.heading: C TRAINING AND VALIDATING DETAILS
sections.text: The network is initialized with ImageNet pretrained weights. During the training phase of 100 epoch, we set the initial learning rate as 0.001 and adopt a linear warm-up strategy to linearly increase the learning rate to 0.01 in the first ten epochs; then, we use a multi-stage learning rate scheduler by multiplying the learning rate by 0.1 in 40th and 70th epoch. We use SGD optimizer with a weight decay of 0.0005 and momentum of 0.9. Considering that the images for vehicle reidentification are usually well-aligned, hence, we use original strided pooling as downsampling in Conv3. As for fine-grained image classification, during the inference phase, the input images are resized to 512×512 then center-cropped to 448×448. Our network is implemented in PyTorch and all experiments have been conducted on one NVIDIA Titan RTX GPU with 24GB memory.