summaries: The authors present a method to jointly learn a shape and UV parameterization of a creased document observed from multiple views. They use an SDF based implicit shape representation in combination with a recent neural volumetric renderer IDR, and optimize the energy composed of the main photometric term (a discrepancy between the observed and rendered image) and regularization terms enforcing well behaved UV space and a fixed scale conformal mapping which corresponds to a behaviour of a real paper material. Since the implicit field representation does not explicitly model a surface, the authors cannot directly optimize the 3D->2D mapping using standard graphics techniques and rather adapt the IDR renderer to be conditioned on the locations sampled from the UV space and train it jointly with the MLPs defined for the bijective mapping 2D->3D and 3D->2D. The authors show that they can get better results both quantitatively and qualitatively when compared to one of the SotA works DewarpNet.

summaries: **High-Level Overview**  This paper proposes a neural rendering technique to learn how to estimate an implicit surface and its UV-parameterization from a set of 2D images. The proposed approach is applied to the problem of document unwarping, wherein a wrinkled, folded, or bent document (e.g. a piece of paper with text on it) can be digitally flattened with its contents preserved. For good measure, the paper also shows that learning the texture mapping enables the contents of the unwarped document to be edited in the texture space. The proposed approach is evaluated against a prior work, DewarpNet (Das et al., 2019) on the basis of Multi-Scale Structural Similarity (MS-SSIM) and Local Distortion (LD) as well as the auxiliary task of OCR.  **Key Contributions**  * Builds on IDR (Yariv et al., 2020) to learn an explicit bijective texture mapping for a learned implicit surface * A loss function formulation for the framework * Application of learned implicit surfaces for document unwarping with reduction in texture distortion and improvements in subsequent OCR metrics  **How It Works**  * Starts with the same problem formulation as IDR: an MLP to model the surface (Eq. 1) and an appearance MLP to model the color (Eq. 2) * Modifies the appearance MLP to also be a function of texture coordinates (Eq. 3) * Adds an additional forward and backward MLP to learn the bijective UV mapping * Jointly trains all MLPs using a pixelwise rendering loss

summaries: This paper presents document unwrapping based on optimization of neural networks. Four MLPs are defined to model the implicit surface of the target, the colors of the target, the mapping function from 3D points to 2D texture coordinates, and the inverse mapping from 2D to 3D. Similarly to NERF, the target example is approximated by training the MLPs using differentiable rendering.

summaries: The paper presents a multi-view method for undistorting documents. The method makes use of an existing method (IDR) for reconstructing distorted documents in 3-D, with an additional bijective neural network to parameterize distorted surfaces by rectangles. The method significantly outperforms SOTAs in the experiments.

abstractText: In this paper, we present a novel approach to learn texture mapping for a 3D surface and apply it to document image unwarping. We propose an efficient method to learn surface parameterization by learning a continuous bijective mapping between 3D surface positions and 2D texture-space coordinates. Our surface parameterization network can be conveniently plugged into a differentiable rendering pipeline and trained using multi-view images and rendering loss. Recent work on differentiable rendering techniques for implicit surfaces has shown high-quality 3D scene reconstruction and view synthesis results. However, these methods typically learn the appearance color as a function of the surface points and lack explicit surface parameterization. Thus they do not allow texture map extraction or texture editing. By introducing explicit surface parameterization and learning with a recent differentiable renderer for implicit surfaces, we demonstrate state-of-the-art document-unwarping via texture extraction. We show that our approach can reconstruct high-frequency textures for arbitrary document shapes in both synthetic and real scenarios. We also demonstrate the usefulness of our system by applying it to document texture editing.

authors:
David Bau
Hendrik Strobelt
William Peebles
Jonas Wulff
Bolei Zhou
Jun-Yan Zhu
Antonio Torralba

title: Semantic photo manipulation with a generative image prior

venue: ACM Transactions on Graphics (TOG),

authors:
Jan Bednarik
Shaifali Parashar
Erhan Gundogdu
Mathieu Salzmann
Pascal Fua

title: Shape reconstruction by learning differentiable surface representations

venue: In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition,

authors:
Alexander W. Bergman
Petr Kellnhofer
Gordon Wetzstein

title: Fast training of neural lumigraph representations using meta learning, 2021

authors:
Sai Bi
Nima Khademi Kalantari
Ravi Ramamoorthi

title: Patch-based optimization for image-based texture mapping

authors:
Caroline Chan
Shiry Ginosar
Tinghui Zhou
Alexei A Efros

title: Everybody dance now

venue: In Proceedings of the International Conference on Computer Vision,

authors:
Tony Chan
Wei Zhu

title: Level set based shape prior segmentation

authors:
Anpei Chen
Zhang Chen
Guli Zhang
Kenny Mitchell
Jingyi Yu

title: Photo-realistic facial details synthesis from single image

authors:
Anpei Chen
Zexiang Xu
Fuqiang Zhao
Xiaoshuai Zhang
Fanbo Xiang
Jingyi Yu
Hao Su

title: MVSNeRF: Fast generalizable radiance field reconstruction from multi-view stereo

venue: arXiv preprint arXiv:2103.15595,

authors:
Zhiqin Chen
Hao Zhang

title: Learning implicit fields for generative shape modeling

venue: In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,

authors:
Christopher B Choy
Danfei Xu
JunYoung Gwak
Kevin Chen
Silvio Savarese

title: 3d-r2n2: A unified approach for single and multi-view 3d object reconstruction

venue: In European conference on computer vision,

authors:
Sagnik Das
Gaurav Mishra
Akshay Sudharshana
Roy Shilkrot

title: The Common Fold: Utilizing the Four-Fold to Dewarp Printed Documents from a Single Image

venue: In Proceedings of the 2017 ACM Symposium on Document Engineering,

authors:
Sagnik Das
Ke Ma
Zhixin Shu
Dimitris Samaras
Roy Shilkrot

title: DewarpNet: Single-image document unwarping with stacked 3D and 2D regression networks

authors:
Sagnik Das
Kunwar Yashraj Singh
Jon Wu
Erhan Bas
Vijay Mahadevan
Rahul Bhotika
Dimitris Samaras

title: End-to-end piece-wise unwarping of document images

venue: In Proceedings of the IEEE/CVF International Conference on Computer Vision,

authors:
Jiankang Deng
Shiyang Cheng
Niannan Xue
Yuxiang Zhou
Stefanos Zafeiriou

title: Uv-gan: Adversarial facial uv map completion for pose-invariant face recognition

authors:
Stephan J. Garbin
Marek Kowalski
Matthew Johnson
Jamie Shotton
Julien Valentin

title: Fastnerf: High-fidelity neural rendering at 200fps, 2021

authors:
Amos Gropp
Lior Yariv
Niv Haim
Matan Atzmon
Yaron Lipman

title: Implicit geometric regularization for learning shapes

venue: arXiv preprint arXiv:2002.10099,

authors:
Thibault Groueix
Matthew Fisher
Vladimir G Kim
Bryan C Russell
Mathieu Aubry

title: A papier-mâché approach to learning 3d surface generation

authors:
Steven Haker
Sigurd Angenent
Allen Tannenbaum
Ron Kikinis
Guillermo Sapiro
Michael Halle

title: Conformal surface parameterization for texture mapping

venue: IEEE Transactions on Visualization and Computer Graphics,

title: Sphere tracing: A geometric method for the antialiased ray tracing of implicit surfaces

venue: The Visual Computer,

authors:
Peter Hedman
Julien Philip
True Price
Jan-Michael Frahm
George Drettakis
Gabriel Brostow

title: Deep blending for free-viewpoint image-based rendering

authors:
Hiroharu Kato
Yoshitaka Ushiku
Tatsuya Harada

title: Neural 3d mesh renderer

authors:
Taeho Kil
Wonkyo Seo
Hyung Il Koo
Nam Ik Cho

title: Robust Document Image Dewarping Method Using Text-Lines and Line Segments

venue: In Proceedings of the International Conference on Document Analysis and Recognition,

authors:
Hyung Il Koo
Jinho Kim
Nam Ik Cho

title: Composition of a dewarped and enhanced document image from two view images

venue: IEEE Transactions on Image Processing,

authors:
Tzu-Mao Li
Miika Aittala
Frédo Durand
Jaakko Lehtinen

title: Differentiable monte carlo ray tracing through edge sampling

authors:
Xiaoyu Li
Bo Zhang
Jing Liao
Pedro V. Sander

title: Document Rectification and Illumination Correction using a Patch-based CNN

authors:
Zhengqi Li
Simon Niklaus
Noah Snavely
Oliver Wang

title: Neural scene flow fields for space-time view synthesis of dynamic scenes

venue: arXiv preprint arXiv:2011.13084,

authors:
Jian Liang
Daniel DeMenthon
David Doermann

title: Geometric rectification of camera-captured document images

venue: IEEE Transactions on Pattern Analysis and Machine Intelligence,

authors:
Ce Liu
Jenny Yuen
Antonio Torralba

title: Sift flow: Dense correspondence across scenes and its applications

authors:
Shichen Liu
Tianye Li
Weikai Chen
Hao Li

title: Soft rasterizer: A differentiable renderer for image-based 3d reasoning

authors:
Shichen Liu
Shunsuke Saito
Weikai Chen
Hao Li

title: Learning to infer implicit surfaces without 3d supervision

venue: arXiv preprint arXiv:1911.00767,

authors:
Ke Ma
Zhixin Shu
Xue Bai
Jue Wang
Dimitris Samaras

title: DocUNet: Document Image Unwarping via A Stacked U-Net

authors:
Amir Markovitz
Inbal Lavi
Or Perel
Shai Mazor
Roee Litman

title: Can you read me now? Content aware rectification using angle supervision

venue: In Proceedings of the European Conference on Computer Vision. Springer,

authors:
Abhimitra Meka
Christian Haene
Rohit Pandey
Michael Zollhöfer
Sean Fanello
Graham Fyffe
Adarsh Kowdle
Xueming Yu
Jay Busch
Jason Dourgarian

title: Deep reflectance fields: Highquality facial reflectance field inference from color gradient illumination

authors:
Gaofeng Meng
Zuming Huang
Yonghong Song
Shiming Xiang
Chunhong Pan

title: Extraction of virtual baselines from distorted document images using curvilinear projection

authors:
Gaofeng Meng
Yuanqi Su
Ying Wu
Shiming Xiang
Chunhong Pan

title: Exploiting Vector Fields for Geometric Rectification of Distorted Document Images

venue: In Proceedings of the European Conference on Computer Vision,

authors:
Lars Mescheder
Michael Oechsle
Michael Niemeyer
Sebastian Nowozin
Andreas Geiger

title: Occupancy networks: Learning 3d reconstruction in function space

authors:
Ben Mildenhall
Pratul P Srinivasan
Matthew Tancik
Jonathan T Barron
Ravi Ramamoorthi
Ren Ng

title: NeRF: Representing scenes as neural radiance fields for view synthesis

authors:
Frederic P. Miller
Agnes F. Vandome
John McBrewster

title: Levenshtein Distance: Information Theory, Computer Science, String (Computer Science), String Metric, Damerau?Levenshtein Distance, Spell Checker

authors:
Aymen Mir
Thiemo Alldieck
Gerard Pons-Moll

title: Learning to transfer texture from clothing images to 3d humans

authors:
Michael Niemeyer
Lars Mescheder
Michael Oechsle
Andreas Geiger

title: Differentiable volumetric rendering: Learning implicit 3d representations without 3d supervision

authors:
Jeong Joon Park
Peter Florence
Julian Straub
Richard Newcombe
Steven Lovegrove

title: Deepsdf: Learning continuous signed distance functions for shape representation

authors:
Taesung Park
Ming-Yu Liu
Ting-Chun Wang
Jun-Yan Zhu

title: Semantic image synthesis with spatially-adaptive normalization

authors:
Chaitanya Patel
Zhouyingcheng Liao
Gerard Pons-Moll

title: Tailornet: Predicting clothing in 3d as a function of human pose, shape and garment style

venue: In IEEE Conference on Computer Vision and Pattern Recognition (CVPR)

authors:
Albert Pumarola
Antonio Agudo
Lorenzo Porzi
Alberto Sanfeliu
Vincent Lepetit
Francesc Moreno-Noguer

title: Geometry-Aware Network for Non-Rigid Shape Prediction from a Single View

authors:
Albert Pumarola
Enric Corona
Gerard Pons-Moll
Francesc Moreno-Noguer

title: D-nerf: Neural radiance fields for dynamic scenes

venue: arXiv preprint arXiv:2011.13961,

authors:
Johannes Lutz Schönberger
Jan-Michael Frahm

title: Structure-from-motion revisited

venue: In Conference on Computer Vision and Pattern Recognition (CVPR),

authors:
Katja Schwarz
Yiyi Liao
Michael Niemeyer
Andreas Geiger. Graf

title: Generative radiance fields for 3d-aware image synthesis

venue: In Advances in Neural Information Processing Systems,

authors:
Vincent Sitzmann
Justus Thies
Felix Heide
Matthias Nießner
Gordon Wetzstein
Michael Zollhofer

title: Deepvoxels: Learning persistent 3d feature embeddings

authors:
Vincent Sitzmann
Michael Zollhöfer
Gordon Wetzstein

title: Scene representation networks: Continuous 3d-structure-aware neural scene representations

venue: arXiv preprint arXiv:1906.01618,

authors:
Vincent Sitzmann
Julien N.P. Martel
Alexander W. Bergman
David B. Lindell
Gordon Wetzstein

title: Implicit neural representations with periodic activation functions

venue: In arXiv,

authors:
Chengzhou Tang
Ping Tan

title: Ba-net: Dense bundle adjustment network

venue: arXiv preprint arXiv:1806.04807,

authors:
Yuandong Tian
Srinivasa G Narasimhan

title: Rectification and 3D reconstruction of curved document images

authors:
Yochay Tzur
Ayellet Tal

title: FlexiStickers: Photogrammetric texture mapping using casual images

venue: In Proceedings of the ACM SIGGRAPH Conference on Computer Graphics,

authors:
Adrian Ulges
Christoph H. Lampert
Thomas Breuel

title: Document Capture Using Stereo Vision

venue: In Proceedings of the 2004 ACM Symposium on Document Engineering,

authors:
Toshikazu Wada
Hiroyuki Ukida
Takashi Matsuyama

title: Shape from shading with interreflections under a proximal light source: Distortion-free copying of an unfolded book

venue: International Journal of Computer Vision,

authors:
Nanyang Wang
Yinda Zhang
Zhuwen Li
Yanwei Fu
Wei Liu
Yu-Gang Jiang

title: Pixel2mesh: Generating 3d mesh models from single rgb images

venue: In Proceedings of the European Conference on Computer Vision (ECCV),

authors:
Zhou Wang
Eero P Simoncelli
Alan C Bovik

title: Multiscale structural similarity for image quality assessment

venue: In The Thirty-Seventh Asilomar Conference on Signals, Systems and Computers,

authors:
Shih-En Wei
Jason Saragih
Tomas Simon
Adam W Harley
Stephen Lombardi
Michal Perdoch
Alexander Hypes
Dawei Wang
Hernan Badino
Yaser Sheikh

title: Vr facial animation via multiview image translation

authors:
Fanbo Xiang
Zexiang Xu
Miloš Hašan
Yannick Hold-Geoffroy
Kalyan Sunkavalli
Hao Su

title: NeuTex: Neural texture mapping for volumetric neural rendering

venue: arXiv preprint arXiv:2103.00762,

authors:
Zexiang Xu
Kalyan Sunkavalli
Sunil Hadap
Ravi Ramamoorthi

title: Deep image-based relighting from optimal sparse samples

authors:
Lior Yariv
Yoni Kasten
Dror Moran
Meirav Galun
Matan Atzmon
Basri Ronen
Yaron Lipman

title: Multiview neural surface reconstruction by disentangling geometry and appearance

venue: Advances in Neural Information Processing Systems,

authors:
Shaodi You
Yasuyuki Matsushita
Sudipta Sinha
Yusuke Bou
Katsushi Ikeuchi

title: Multiview Rectification of Folded Documents

authors:
Alex Yu
Vickie Ye
Matthew Tancik
Angjoo Kanazawa

title: pixelNeRF: Neural radiance fields from one or few images

venue: arXiv preprint arXiv:2012.02190,

authors:
Fang Zhao
Shengcai Liao
Kaihao Zhang
Ling Shao

title: Human parsing based texture transfer from single image to 3D human via cross-view consistency

heading: 1 INTRODUCTION

text:
Reconstructing 3D shapes from images is a core problem in computer vision and graphics research. With the progress in differentiable rendering (Sitzmann et al., 2019b; Kato et al., 2018; Niemeyer et al., 2020; Li et al., 2018; Liu et al., 2019b), recent learning-based 3D reconstruction approaches have achieved impressive results using 2D supervision from single image (Chen & Zhang, 2019; Groueix et al., 2018; Choy et al., 2016; Mescheder et al., 2019; Wang et al., 2018) or multi-view images (Tang & Tan, 2018; Yariv et al., 2020). These methods achieve high quality 3D reconstruction using differentiable rendering with various 3D representations such as 3D mesh (Wang et al., 2018), volumetric representation (Mildenhall et al., 2020), or implicit functions (Mescheder et al., 2019). In recent neural rendering methods such as NeRF (Mildenhall et al., 2020) and IDR (Yariv et al., 2020), continuous representations such as volume or implicit functions achieve significantly better reconstruction results than meshes or voxels because they do not discretize the 3D surface a priori. However, these continuous representations usually do not encode explicit surface parameterization, allowing 3D shape re-texturing, editing the existing texture in the 2D texture space, or recovering 2D texture from 3D surfaces. One of the most direct applications of 2D texture recovery in a geometrically constrained manner, is document unwarping, i.e., inference of a document’s flatbed-scanned version from a casual photo of a potentially creased document. Whereas 2D texture recovery could be equally valuable for other domains such as garments, or faces, the existing datasets are not directly applicable to our method.
Our novel texture mapping approach learns surface parameterization for document unwarping by learning continuous bijective functions between 3D surface positions and 2D texture-space coordinates. We use a signed distance function (SDF) (Chan & Zhu, 2005) to represent geometry and model the appearance as a function of the 2D texture coordinates. By utilizing implicit differentiable rendering (IDR), (Yariv et al., 2020) we can reconstruct 3D shape and learn the corresponding UV parameterization of the surface simultaneously using a per-pixel rendering loss and appropriate geometric regularizations.
We utilize two fully connected multi-layer perceptrons (MLPs) to learn a bijective mapping between 3D shapes and 2D texture space. More specifically, the forward MLP maps the 3D surface
coordinates to 2D texture coordinates and the backward MLP maps the 2D texture coordinates to corresponding 3D surface coordinates. Following IDR (Yariv et al., 2020), we obtain the 3D surface coordinates by sphere-tracing along the ray, cast through each pixel. Our appearance rendering is formulated as a function of the 3D and the texture coordinates. Therefore, the forward and backward MLPs can be trained with a 2D pixel-wise loss between the rendered image and the given ground truth image. To the best of our knowledge, this is the first neural rendering method that can learn effective UV parameterization for implicit surfaces.
As a corollary, our method is also the first method which utilizes implicit surface based neural rendering for document unwarping. It is a challenging task due to the presence of geometric and photometric distortions in the document. For this particular problem we introduce a prior for shapespecific texture mapping to initialize the forward MLP (3D to 2D mapping). This prior is learned from a large dataset of UV mapped document meshes, assuming that document texture space maps to a 2D equiangular quadrilateral. This assumption regularizes the forward MLP to output a highquality texture space that avoids degenerate solutions (see Fig. 3). Moreover, we introduce a conformality constraint in the backward MLP, which is consistent with how paper folds happen in the physical world, i.e., without any stretch or tear. This constraint also ensures that the backward function is bijective and smooth (Petrini et al., 2018).
The main contributions of our paper are the following: First, we propose an efficient way to learn texture parameterization for implicit neural representations using a differentiable rendering framework. Without 3D supervision, it only requires multi-view images as ground-truth and a texture mapping prior. Second, we show that our method can be effectively used for document unwarping tasks by learning a prior for explicit texture mapping on the document shape. We show that this prior can be learned from a dataset of texture-mapped meshes. Third, we show that our method is effective for document image unwarping and texture editing (see Fig. 1). We achieve a 52% relative improvement over the publicly available state-of-the-art1 (Das et al., 2019) in terms of mean local distortion across 500 views from ten synthetic scenes. Additionally, we achieve a ∼25% improvement in optical character recognition (OCR) in terms of character and word error rate.

heading: 2 PREVIOUS WORK

text:
Neural Rendering. Neural rendering generates images and videos by integrating conventional computer graphics rendering pipelines into deep neural networks (Tewari et al., 2020). It enables explicit or implicit control of scene properties, including illumination, geometry, texture, etc. Neural rendering can synthesize semantic photos (Park et al., 2019b; Bau et al., 2019), novel views (Hedman et al., 2018; Sitzmann et al., 2019a), relighting (Xu et al., 2018; Meka et al., 2019), facial/body reenactment (Chan et al., 2019; Wei et al., 2019), estimate scene properties etc. Kato (Kato et al., 2018) proposed a differentiable neural renderer using an approximate gradient for rasterization. Liu (Liu et al., 2019a) proposed SoftRas, which extended differentiable rasterization. Li (Li et al., 2018) further demonstrated the feasibility of integrating ray-tracing in deep neural networks. More recently, implicit surface or volume rendering has become mainstream in neural rendering approaches such as IDR (Yariv et al., 2020) and NeRF (Mildenhall et al., 2020). These approaches are based
1A more recent approach, CREASE (Markovitz et al., 2020), data and models are not publicly available.
on multi-view surface reconstruction to associate the scene geometry to the appearance in different views. NeRF is extended to lot of variants including PixelNeRF (Yu et al., 2020), MVSNeRF (Chen et al., 2021), dynamic NeRF (Li et al., 2020; Pumarola et al., 2020), GRAF (Schwarz et al., 2020) and so on.
Texture Mapping. Texture mapping is an essential step in the computer graphics rendering pipeline. It defines a correspondence between a vertex on the 3D mesh and a pixel in the 2D texture image. To find such a mapping, FlexiStickers (Tzur & Tal, 2009) required users to specify a sparse set of correspondences. Bi (Bi et al., 2017) proposed a patch-based texture mapping method using the 3D shape and images from multiple views. Morreale (Morreale et al., 2021) used networks to represent 3D surfaces/shapes. Besides the above general texture mapping methods, some approaches focus on a specific object categories such as faces (Deng et al., 2018; Chen et al., 2019) and human bodies (Mir et al., 2020; Zhao et al., 2020). Recently, AtlasNet (Groueix et al., 2018) represented a 3D mesh as a collection of parametric surfaces; thus, texture mapping is trivial to obtain from a 2D parametric surface. A similar idea was adopted by Bednarik (Bednarik et al., 2020) where they introduced geometric constraints when learning the decomposition. More recently NeuTex (Xiang et al., 2021) aims to recover the texture of a subject using NeRF (Mildenhall et al., 2020). However, NeuTex uses a spherical UV domain without any geometric constraints. Therefore, it is not suitable for document unwarping. Moreover, since NeRF (Mildenhall et al., 2020) doesn’t learn an explicit geometry, NeuTex requires a coarse point-cloud to initialize the backward MLP. With an SDF based (Yariv et al., 2020) approach, our approach does not require such an initialization routine. We jointly learn the texture mapping and the geometry from scratch.
Document Unwarping. Document unwarping is a special application of texture mapping: the 3D object is usually a rectangular piecewise-developable surface, and the texture is well-structured, containing straight text lines, (usually) rectangular text blocks and figures, etc. Previous work usually adopted a two-step methodology: 1) 3D surface estimation and 2) deformed surface flattening. The 3D surface of a deformed document can be estimated from shading (Wada et al., 1997), multi-view images (Ulges et al., 2004), text lines (Tian & Narasimhan, 2011), local character orientations (Meng et al., 2018), document boundaries (Koo et al., 2009), and learning-based strategies (Pumarola et al., 2018). Flattening the obtained 3D surface always involves an expensive optimization process under certain geometry constraints such as conformality (You et al., 2017) or isometries (Bartoli et al., 2015). Flattening could be easier if the obtained 3D shape had a low dimensional parameterization like Generalized Cylindrical Surface (GCS) (Kil et al., 2017). Some studies (Das et al., 2017; Liang et al., 2008; Meng et al., 2015) proposed to unwarp each patch on the surface individually and then stitch the unwarped patches together. In recent years, data-driven methods (Ma et al., 2018; Das et al., 2019; Li et al., 2019; Markovitz et al., 2020; Das et al., 2021) have addressed document unwarping by leveraging large-scale synthetic datasets. These datasets contain deformed document images and their corresponding ground-truth UV coordinates. Methods trained on synthetic images often suffer from generalization performance due to the domain gap between synthetic and real data. In this paper, we utilize neural rendering techniques to learn a surface parameterization of a deformed document. We simultaneously estimate both 3D shapes and UV coordinates with a cycle consistency loss and geometric constraints. By leveraging the information from multi-view images, the proposed method demonstrates better document unwarping performance compared to a previous state-of-the-art, Das et al. (2019).

heading: 3 METHOD

text: A schematic diagram of the proposed approach is shown in Fig. 2. We utilize a recent differentiable rendering method, IDR (Yariv et al., 2020) for surface reconstruction and jointly learn the texture mapping of the learned implicit surface using two MLPs. In Sec. 3.1 we first describe some preliminaries about surface parameterization and IDR.

heading: 3.1 PRELIMINARIES

text:
Surface Parameterization. The problem of surface parameterization focuses on finding a bijective mapping F between a surface Z ∈ R3 and a polygonal domain Ω ∈ Rn. For a parametric or discrete surface representation, we can explicitly compute this mapping (Tzur & Tal, 2009) using constrained optimization. In contrast, implicit surfaces are represented as continuous functions and cannot be readily parameterized. In this paper, we propose to learn such bijective mapping between a learned
implicit surface and a 2D planar domain Ω ∈ R2 using our proposed forward and backward MLPs. Ω is the texture space or UV space, parameterized using 2D UV coordinates t = (u, v). We can use any continuous parameterization function as the UV space. Since this work particularly focuses on document unwarping, we choose the UV space to be a regular 2D grid.
Implicit Differentiable Rendering. Implicit Differentiable Rendering (Yariv et al., 2020) reconstructs the geometry of an object from multi-view images as the zero level set, Zθ of an MLP S,
Zθ = {z ∈ R3 | S(z; θ) = 0} (1) where θ are the learnable parameters. To render the surface Zθ, IDR uses another MLP to model the radiance (RGB color) as a function of the surface point (zp), corresponding surface normal (np), view direction (vp) and a global geometry feature vector (gp):
Cp = A(zp,np, vp, gp) (2) Here, Cp denote the predicted color at pixel p and A denotes the appearance MLP. The surface point is obtained by a sphere-tracing method (Hart, 1996) along the ray rp(τ) through pixel p. τ ∈ Rk denotes camera parameters of the scene. Additionally, IDR also presents a differentiable way to obtain a ray and geometry intersection point (ẑp) as a function of the camera ray. Although,the IDR can disentangle geometry and appearance, it only allows to re-texture a new geometry with a learned appearance MLP, A. Editing a texture or extracting a surface texture map is not possible in a vanilla IDR framework since no explicit texture mapping is learned.

heading: 3.2 LEARNING SURFACE PARAMETERIZATION

text:
To learn a meaningful parameterization of the implicit surface Zθ, we represent the radiance at pixel p as a function of the UV space. To this end, we modify the IDR model (Eq. 2):
Cp = Auv(tp, zp,np, vp, gp) (3) The texture parameterized appearance MLP is modeled as a function of the texture coordinate tp at surface point zp, corresponding to a pixel p. We can jointly train the surface MLP (S) and texture parameterized appearance MLP (Auv) using a pixel wise rendering loss between the predicted radiance (Cp) and ground-truth radiance (Cgtp ) at pixel p.
Forward and backward texture parameterization. We represent the mapping between the 3D surface and 2D texture space using the forward function Fuv:
z→ t. (4) The Fuv is modeled as an MLP. It is trained by mapping a ray-surface intersection point ẑp to its corresponding texture coordinate tp corresponding to a pixel p. Now to establish the bijective mapping (discussed in Sec. 3.1) between the surface and texture space we utilize a backward function Fz: t→ z. (5) Fz is an MLP that learns an inverse mapping between the texture and the 3D space. It is trained by mapping a texture coordinate tp to its corresponding ray-surface intersection point ẑp.
Shape specific prior for Fuv . Jointly training the forward, backward and rendering network leads
to the wrong UV mapping with local minima (see Fig. 3) where multiple ẑp map to a single texture coordinate. To avoid such degenerate cases, we initialize Fuv with a texture mapping prior, learned from a large dataset of UV mapped meshes. This learned prior (F̂uv) makes the learned texture mapping suitable for document unwarping. We assume the document shape to be a deformed quadrilateral and the corresponding UV space to be a regular grid (∈ [0.0, 1.0]). The top leftmost and the bottom rightmost 3D coordinate of the shape maps to (u, v) = (0, 0) and (u, v) = (1, 1) respectively. To learn F̂uv we utilize a collection of UV mapped document meshes from the Doc3D (Das et al., 2019) dataset and train an MLP with the same parameters as Fuv . For each scene, we use F̂uv to initialize the weights of Fuv and train jointly with S and Auv .
Deformation constraints for Fz. Conformal map (Haker et al., 2000) allows a 3D domain to be mapped to a texture domain with low distortion satisfying the bijective property between domains. We use a conformality constraint for Fz to ensure the deformation properties mentioned above. We define the conformality constraint in terms of the metric tensor, J>J of the Fz, where J is the jacobian of Fz (Eq. 6):
J = [ δFz δu δFz δv ] = [Du Dv] J>J = [ D>uDu D > uDv
D>uDv D > v Dv
] = [ E F
F G
] (6)
The conformality constraint is defined as J>J = βI. Here β is a unknown local scaling function and I is the identity matrix. For developable surfaces which can be physically flattened without any stretch e.g. papers, β doesn’t vary across the parameterization space. Therefore, we consider a fixed global scale ([βgu, β g v ]) for the conformality constraint.
Unwarping by sampling Fz. To unwarp an input image, we determine the pixel at p = (x, y) in the input image should be projected to (u, v) in the unwarped image. Here the unwarped image refers to the texture space. The coordinates (u, v) and p are associated by Fz and τ : For a (u, v) coordinate, its corresponding point in 3D is obtained by ẑ′p = Fz(u, v). Given the camera parameter τ , ẑ′p is projected to p in the input image. Thus for each pixel in the unwarped image, we can find its corresponding pixel in the input image which is all we need for unwarping.

heading: 3.3 LOSS FUNCTIONS

text:
We use the rendering losses on the predicted color, Cp, and predicted document mask Mp at pixel p to train the geometry S. Here Mp ∈ {0, 1} refers to whether the pixel p is occupied (Mp = 1) by the shape or not (Mp = 0). We assume masks are provided as input. Additionally, we employ appropriate regularization losses to jointly train S, Auv , Fuv and Fz .
Loss for S. Following IDR (Yariv et al., 2020), for each p we apply a sphere tracing (Hart, 1996) algorithm to find the intersection point of the ray rp(τ) and the surface Zθ. Given the ground-truth RGB color Cgtp and the predicted RGB color Cp, the RGB loss is defined as:
Lrgb = 1 |P | ∑ p∈Pin ∥∥Cgtp − Cp∥∥1 (7) Where P is the set of pixels in the minibatch. The pixels Pin ⊂ P for which ray surface intersection has been found and Mp = 1. The mask loss is defined as:
Lmask = 1 α|P | ∑
p∈Pout
CE(Mgtp ,Mp) (8)
Here Pout = P \Pin, alpha is a tunable parameter and CE(.) is the cross-entropy loss. The value of Mp =Mp,α(θ, τ) is a differentiable function of the learned Zθ (Yariv et al., 2020). Additionally, to force Zθ to be a approximate signed distance function we use Eikonal Regularization (Gropp et al.,
2020): Lek = Ez(‖∇zS(z; θ)‖ − 1)2 (9) where z denotes uniformly sampled points within a bounding box of the 3D domain.
Loss for Fuv. Although we initialize Fuv with learned prior parameters, we constrain the predicted 2D texture coordinates during training in order to avoid non-uniform mapping of the 3D and the UV domain which can squeeze or stretch the warped texture (example in supplementary). We employ a Chamfer distance between the tp and uniformly sampled 2D points T ∈ [0, 1] to ensure Fuv approximately outputs U ∼ [0, 1]. This regularization term is defined as:
Luv = CDp∈Pin(T , tp) (10) here CD(.) denotes the Chamfer distance and tp the predicted texture coordinates corresponding to ray-surface intersection points ẑp.
Loss for Fz. ẑ′p is the output of Fz. Fz is trained with weighted regression loss between ẑp and ẑ′p:
Lz = 1 |Pin| ∑ p∈Pin wp(ẑp − ẑ′p)2 (11)
wp is a pre-calculated per-pixel weight based on the document mask (M ) which assigns higher value to the pixels at the boundary of the document. (More weight calculation details in Supplementary).
Additionally, to constrain Fz to be a fixed scale conformal mapping (Bednarik et al., 2020). We employ three constraints on the elements of the metric tensor E, F and G defined in Eq. 6.
LE = 1 |Pin| ∑ p∈Pin (Ep − Ẽ)2 LG = 1 |Pin| ∑ p∈Pin (Gp − G̃)2 LF = 1 |Pin| ∑ p∈Pin (Fp) 2
Here Ẽ and G̃ is the mean of E and G.
Our combined loss function is defined as: L = (Lrgb + γ1Lmask + γ2Lek)︸ ︷︷ ︸
LS
+ρLuv + (δ1Lz + δ2LE + δ3LG + δ4LF )︸ ︷︷ ︸ LT
(12)
Here γ, ρ and δ denote the hyperparameters associated with the losses.

heading: 3.4 TRAINING DETAILS

text: The surface MLP S(z, θ) consists of 8 layers with a hidden layer dimension of 128, with a skip connection to the middle layer (Park et al., 2019a). Following IDR (Yariv et al., 2020), S is initialized to produce an approximate SDF of a unit sphere. The rendering network Auv has 4 layers with hidden layer dimension of 512 and uses a sine activation function (Sitzmann et al., 2020) at each layer. Fuv and Fz share identical architecture with 8 layers with 512 dimensional hidden units and sine activation (Sitzmann et al., 2020). Following NeRF (Mildenhall et al., 2020), we use a k dimensional Fourier mapping (χk : R → R2k) to learn high frequency details in the shape, RGB and the UV space. For S, Auv we follow the setting of (Yariv et al., 2020), and set k = 6 and k = 4 respectively. For Fuv and Fz we empirically set number of Fourier bands k = 10. We start with an initial learning rate of 1e×−5 and train for 150K iterations by halving the learning rate after every 50K iterations. Initially, α is set to 50 and doubled during the training after every 50K iterations. We set γ1 = 100.0, γ2 = 0.1 and ρ = 0.001. δ1 is set to 0.001 for the initial 30K iterations. Afterward, δ1 is multiplied by a factor 2 at every 10K iterations for a maximum of 7 times. δ2, δ3 and δ4, are set to zero for the initial 100K iterations. Only Lz is sufficient to achieve a good texture to 3D mapping during the shape optimization phase. Afterwards we set δ2 = δ3 = 0.001 and δ4 = 0.01. The metric tensor calculation is implemented using auto-differentiation.

heading: 4 EXPERIMENTAL RESULTS

text: First, we quantitatively compare the proposed method with state-of-the-art document unwarping method DewarpNet (Das et al., 2019). Our quantitative and qualitative experiments are performed on 10 synthetic scenes and 10 real scenes. Second, we apply our method to texture editing. Last, we conduct ablation studies to demonstrate the effectiveness of our proposed loss functions.

heading: 4.1 EVALUATION DATASET AND METRICS

text:
Our synthetic evaluation data consists of 10 scenes rendered using Blender following a rendering pipeline similar to Doc3D. Each scene consists of 50 random views sampled from a 45o solid angle in the upper hemisphere. The real-world evaluation data consists of 3 scenes from the dataset of (You et al., 2017), and 9 scenes captured by us. Each scene consists of 5-20 images per scene. We manually annotate the masks for each scene. To obtain camera poses for the real-world data, we utilize the COLMAP (Schönberger & Frahm, 2016) multi-view reconstruction pipeline. Both synthetic and real data include the document scan, as the unwarping ground-truth.
We use image-based evaluation metrics for quantitative evaluation, including Local Distortion (LD) and Multi-Scale Structural Similarity (MS-SSIM). These are standard metrics used for document unwarping evaluation (Das et al., 2019; Ma et al., 2018). LD is based on dense SIFT flow (Liu et al., 2011) between the unwarped and scanned images. Image similarity metric, MS-SSIM (Wang et al., 2003) is based on local image statistics (mean and variance) of the unwarped and scanned (groundtruth) images calculated over multiple Gaussian pyramid scales. We use the same settings as (Das et al., 2019; Ma et al., 2018) for fair comparison.

heading: 4.2 DOCUMENT UNWARPING

text:
The primary application of our learned forward and backward MLP is document unwarping. The quantitative comparison with the state-ofthe-art model (Das et al., 2019) is shown in Table 1 for the synthetic and real scenes. In terms of average performance of all the views (all views col. in Table 1) we improve the LD by ∼ 52% compared to (Das et al., 2019). Since we use multi-view images for training, our results are more consistent across all the views compared to DewarpNet, which is also a key reason for the significant improvement. We conjecture that (Das et al., 2019) as a single image unwarping method should perform well on simpler deformations and frontal view images. However, it is not always the case. In qualitative comparisons in Fig. 4, DewarpNet often gen-
erates artifacts even for reasonably frontal views and simple deformations. Comparatively, our results are qualitatively superior.
We also report in a stricter evaluation scenario (best view column of Table 1) where we compare our results with the best possible numerical results achieved by DewarpNet from a single view in a scene. We perform better than DewarpNet in 91.2% of all views, however when the best view can be selected our method do slightly worse in 7 scenes. This ‘stricter’ setting shows quantitatively competitive results compared to DewarpNet with a oracle (practically challenging) view selector.The choice of the best unwarped result is often subjective. For a more comprehensive comparison, we qualitatively compare the best results of DewarpNet with our results across 6 scenes in Fig. 5. These 6 scenes are chosen among the 7 scenes for which DewarpNet achieves a better quantitative result than the proposed approach for at least one view. In Fig. 5 our results are clearly better than the DewarpNet in all cases, with straighter lines and better rectified structure. The evaluation scores do not accurately reflect the improvement due to the sensitivity of LD and MSSIM to subtle perceptually unimportant global transformations, such as translation of the image by few pixels. However, such transformations do not affect the visual quality or readability of the unwarped results. More discussion and qualitative comparison is available in supplementary material.
DewarpNet Proposed
ED ↓ CER (std) ↓ WER (std) ↓ ED ↓ CER (std) ↓ WER (std) ↓ Mean 798.30 0.2827 (0.12) 0.4646 (0.17) 600.78 0.2122 (0.10) 0.3568 (0.11)
Table 2: Comparison of OCR error metrics: We improve the OCR performance of Das et al. (2019) by∼25% in terms of Edit Distance (ED), Character Error Rate (CER), and Word Error Rate (WER).
The quantitative comparison for real scenes are reported in Table 1 (bottom). We achieve better results in terms of mean and best evaluation score than DewarpNet in 2 out of 3 scenes. We notice that the evaluation results are a little worse for the real scenes than synthetic scenes due to the fewer available views (5-10 compared to 50). Moreover, there are cases like Real 6, which do not have sufficient texture. Such data are a failure case of IDR since there is insufficient information to reconstruct the 3D shape. As a result of the poor 3D shape, our texture parameterization network produces an inferior unwarping result (More details are available in Supplementary). We also report qualitative comparisons with You et al. (2017) and Das et al. (2019) on additional real documents in supplementary.
OCR Evaluation. We also evaluated the OCR performance on 5 real scenes across 77 images in Table 2. We use Edit Distance (ED) (Miller et al., 2009), Character Error Rate (CER) and Word Error Rate (WER) as our evaluation metrics. ED is defined as the total number of substitutions (s), insertions (i) and deletions (d) required to obtain the reference text, given the recognized text. The reference text is obtained by running the OCR algorithm on the scanned ground-truth image of each document. CER is defined as: (s+ i+ d)/N where N is the number of characters in the reference text. We use Tesseract 4.1.1 based LSTM OCR engine for this experiment. Our unwarped results reduce the ED, CER and WER by ∼25%. This improvement proves our unwarped results are more suitable for downstream applications tasks like OCR.
Texture Editing. In addition to document unwarping, our proposed forward and backward MLP can also be used for high quality texture editing. We show two texture editing examples in Fig. 1. We use the backward MLP to unwarp the texture from the input image, then we edit the texture and warp it back to image space using the learned forward MLP. (More details in Supplementary).
Ablation Study. We ablate how loss terms Lz , LE , LF , and LG affect the unwarping results. We train Fuv and Fz with different combinations of these loss terms and report the mean MSSIM and LD in Table 3 (appendix). Qualitative results for one scene are shown in Fig. 6 (appendix).

heading: 5 TRAINING TIME, GENERALIZABILITY AND FUTURE WORK

text:
Our proposed method for a scene can be trained in approximately 18 hours for 448× 448 resolution images using a single Titan Xp GPU. The current training time per scene is very high compared to DewarpNet’s inference time which makes it unsuitable for real time applications. However, this is a fast growing field and there are multiple other works that are focusing on improving the speed and generalization abilities (Garbin et al., 2021; Bergman et al., 2021) of neural rendering. Therefore, obtaining a faster training scheme is considered as a future work.
Our method can be applied to fabrics, which are very similar to papers and lead to practical applications of texture editing. However, none of the current 3D garment/fabric datasets (Patel et al., 2020) can be easily adapted to train the F̂uv prior. For more complex UV spaces (e.g., texture atlas), learning the prior may require decomposing the shape to multiple simple UV maps. The proper way to do this is beyond the scope of this paper, however we believe it’s an exciting future work. As importantly, in this paper, we have introduced a number of domain specific strong constraints that suit the rectangular paper shape. These constraints improve empirical results. More general objects will require different constraints e.g., spherical UV domain, local scaling of the conformal map etc.

heading: 6 CONCLUSIONS

text: We have introduced an end-to-end trainable architecture that can simultaneously learn texture parameterized 3D shapes from multi-view images. This is the first work to learn surface parameterization of an implicit neural representation to the best of our knowledge. We have demonstrated the applicability of our approach on multiple synthetic and real scenes for the task of document unwarping and document texture editing. We want to extend this method to learn surface parameterization for more complex shapes such as faces or general 3D objects in future work.

heading: 7 ETHICS STATEMENT

text: Texture editing application of our proposed approach can have both positive and negative societal impact. On the positive side, real document images can be gracefully redacted to protect sensitive information. On the contrary, it can be potentially used for editing real documents and change the content to commit fraud and spread misinformation.

heading: 8 REPRODUCIBILITY STATEMENT

text: We believe our results are reproducible by following the training details in Section 3.4 of main submission and Section 2, 3, 4, 5 from the supplementary material.

heading: A APPENDIX

text:
A.1 ABLATION STUDY
We ablate how loss terms Lz , LE , LF , and LG affect the unwarping results. We train Fuv and Fz with different combinations of these loss terms and report the mean MSSIM and LD in Table 3. Qualitative results for one scene are shown in Fig. 6 (appendix). In the basic version (listed as b), no conformality constraints (LE , LF and LG) are used and wp = 1 in Lz . The b+w version introduces a weighting function that assigns a higher value to wp if a pixel is closer to the document boundary. Introducing this loss improves the boundary; notice the white margin at the top and bottom in the second column of Fig. 6. Introducing the conformality constraints alleviates the unusual stretch in the texture and improves smoothness (Fig. 6, col. 3). Using both improves the boundary and the texture smoothness (Fig. 6, col. 4) and achieves the best result.
(a) (b) (c)
(d) (e) (f)


summaries: The paper is tackling the topic of fine-grained representation. In contrast to comparable methods, ARP focus on sub-sampled via strided operation to represent fine-grained information.  ARP is an intuitive strategy to extract critical regions and is more concise compared with multi-stage cascade architecture. This fact facilitates the efficiency.

summaries: - **Motivation**. The paper argues that    - (a) there are some small but important regions of an interesting object, which are usually ignored by the previous fine-grained methods;   - (b) excessive reduction operations on image resolution fade the discriminative features.     - **Method**. According to the above observations, the paper proposes a pooling algorithm called Adaptive Region Pooling (ARP). This module has two procedures:   - (a) learn to crop image regions;   - (b) downsampling various size regions using bilinear operation.   - **Experiments**. The proposed method is verified in two tasks: fine-grained classification and re-identification.

summaries: This paper presents an adaptive region pooling (ARP) method for fine-grained representation learning. ARP crops the features with the estimated scale. The features are further downsampled to a consistent size through bilinear downsampling. Experiments conducted on two tasks validate the effectiveness of the proposed method.

summaries: This paper proposes an adaptive region pooling (ARP) module that adaptively estimates the scale factors and crops the discriminative regions based on the estimated scale factors. It aims to focus on the most discriminative region and simultaneously contain more fine-grained information. The authors apply the ARP module to fine-grained image recognition and Vehicle Re-Id tasks.

abstractText: Fine-grained recognition aims to discriminate the sub-categories of the images within one general category. It is fundamentally difficult due to the requirement to extract fine-grained features from subtle regions. Nonetheless, a Convolutional Neural Network typically applies strided operations to downsample the representation, which would excessively spoil the feature resolution and lead to a significant loss of fine-grained information. In this paper, we propose Adaptive Region Pooling (ARP): a novel downsampling algorithm that makes the network only focus on a smaller but more critical region, and simultaneously increase the resolution of sub-sampled feature. ARP owns a trade-off mechanism that allows users to actively balance the scale of receptive field and the granularity of feature. Also, without any learning-based parameters, ARP provides the network a stabler training process and an earlier convergence. Extensive experiments qualitatively and quantitatively validate the effectiveness and efficiency of the proposed pooling operation and show superior performance against the state-of-the-arts in both the tasks of image classification and image retrieval.

authors:
Binghui Chen
Weihong Deng

title: Hybrid-attention based decoupled metric learning for zero-shot image retrieval

venue: In IEEE Conference on Computer Vision and Pattern Recognition,

authors:
Long Chen
Hanwang Zhang
Jun Xiao
Liqiang Nie
Jian Shao
Wei Liu
Tat-Seng Chua

title: Scacnn: Spatial and channel-wise attention in convolutional networks for image captioning

authors:
Tsai-Shien Chen
Man-Yu Lee
Chih-Ting Liu
Shao-Yi Chien

title: Viewpoint-aware channel-wise attentive network for vehicle re-identification

venue: In IEEE Conference on Computer Vision and Pattern Recognition Workshops,

authors:
Tsai-Shien Chen
Chih-Ting Liu
Chih-Wei Wu
Shao-Yi Chien

title: Orientation-aware vehicle re-identification with semantics-guided part attention network

venue: In European Conference on Computer Vision, 2020b

authors:
Yao Ding
Yanzhao Zhou
Yi Zhu
Qixiang Ye
Jianbin Jiao

title: Selective sparse sampling for fine-grained image recognition

venue: In IEEE International Conference on Computer Vision,

authors:
Jianlong Fu
Heliang Zheng
Tao Mei

title: Look closer to see better: Recurrent attention convolutional neural network for fine-grained image recognition

authors:
Bing He
Jia Li
Yifan Zhao
Yonghong Tian

title: Part-regularized near-duplicate vehicle reidentification

authors:
Kaiming He
Xiangyu Zhang
Shaoqing Ren
Jian Sun

title: Deep residual learning for image recognition

authors:
Alexander Hermans
Lucas Beyer
Bastian Leibe

title: In defense of the triplet loss for person reidentification

venue: arXiv preprint arXiv:1703.07737,

authors:
Shaoli Huang
Zhe Xu
Dacheng Tao
Ya Zhang

title: Part-stacked cnn for fine-grained visual categorization

authors:
Max Jaderberg
Karen Simonyan
Andrew Zisserman

title: Spatial transformer networks

authors:
Ruyi Ji
Longyin Wen
Libo Zhang
Dawei Du
Yanjun Wu
Chen Zhao
Xianglong Liu
Feiyue Huang

title: Attention convolutional binary neural tree for fine-grained visual categorization

authors:
Mahdi M Kalayeh
Emrah Basaran
Muhittin Gökmen
Mustafa E Kamasak
Mubarak Shah

title: Human semantic parsing for person re-identification

authors:
Pirazh Khorramshahi
Amit Kumar
Neehar Peri
Sai Saketh Rambhatla
Jun-Cheng Chen
Rama Chellappa

title: A dual-path model with adaptive attention for vehicle re-identification

authors:
Pirazh Khorramshahi
Neehar Peri
Jun-cheng Chen
Rama Chellappa

title: The devil is in the details: Self-supervised attention for vehicle re-identification

venue: arXiv preprint arXiv:2004.06271,

authors:
Jin-Hwa Kim
Jaehyun Jun
Byoung-Tak Zhang

title: Bilinear attention networks

authors:
Jonathan Krause
Michael Stark
Jia Deng
Li Fei-Fei

title: 3d object representations for fine-grained categorization

venue: In IEEE International Conference on Computer Vision Workshops,

authors:
Yann LeCun
Léon Bottou
Yoshua Bengio
Patrick Haffner

title: Gradient-based learning applied to document recognition

venue: Proceedings of the IEEE,

authors:
Di Lin
Xiaoyong Shen
Cewu Lu
Jiaya Jia

title: Deep lac: Deep localization, alignment and classification for fine-grained recognition

authors:
Tsung-Yu Lin
Aruni RoyChowdhury
Subhransu Maji

title: Bilinear cnn models for fine-grained visual recognition

authors:
Chih-Ting Liu
Man-Yu Lee
Tsai-Shien Chen
Shao-Yi Chien

title: Hard samples rectification for unsupervised cross-domain person re-identification

venue: arXiv preprint arXiv:2106.07204,

authors:
Hongye Liu
Yonghong Tian
Yaowei Wang
Lu Pang
Tiejun Huang

title: Deep relative distance learning: Tell the difference between similar vehicles

authors:
Xiaobin Liu
Shiliang Zhang
Qingming Huang
Wen Gao

title: Ram: a region-aware deep model for vehicle re-identification

venue: In IEEE International Conference on Multimedia and Expo,

authors:
Xinchen Liu
Wu Liu
Huadong Ma
Huiyuan Fu

title: Large-scale vehicle re-identification in urban surveillance videos

authors:
Xinchen Liu
Wu Liu
Tao Mei
Huadong Ma

title: A deep learning-based approach to progressive vehicle re-identification for urban surveillance

venue: In European Conference on Computer Vision, 2016c

authors:
Yihang Lou
Yan Bai
Jun Liu
Shiqi Wang
Ling-Yu Duan

title: Embedding adversarial learning for vehicle re-identification

venue: IEEE Transactions on Image Processing,

authors:
Hao Luo
Youzhi Gu
Xingyu Liao
Shenqi Lai
Wei Jiang

title: Bag of tricks and a strong baseline for deep person re-identification

authors:
Wei Luo
Xitong Yang
Xianjie Mo
Yuheng Lu
Larry S Davis
Jun Li
Jian Yang
Ser-Nam Lim

title: Cross-x learning for fine-grained visual categorization

authors:
Subhransu Maji
Esa Rahtu
Juho Kannala
Matthew Blaschko
Andrea Vedaldi

title: Fine-grained visual classification of aircraft

venue: arXiv preprint arXiv:1306.5151,

authors:
Dechao Meng
Liang Li
Xuejing Liu
Yadong Li
Shijie Yang
Zheng-Jun Zha
Xingyu Gao
Shuhui Wang
Qingming Huang

title: Parsing-based view-aware embedding network for vehicle reidentification

authors:
Yuval Netzer
Tao Wang
Adam Coates
Alessandro Bissacco
Bo Wu
Andrew Y Ng

title: Reading digits in natural images with unsupervised feature learning

authors:
Ergys Ristani
Francesco Solera
Roger Zou
Rita Cucchiara
Carlo Tomasi

title: Performance measures and a data set for multi-target, multi-camera tracking

venue: In ECCV Workshop on Benchmarking Multi-Target Tracking,

authors:
Karen Simonyan
Andrew Zisserman

title: Very deep convolutional networks for large-scale image recognition

venue: arXiv preprint arXiv:1409.1556,

authors:
Yifan Sun
Liang Zheng
Yi Yang
Qi Tian
Shengjin Wang

title: Beyond part models: Person retrieval with refined part pooling (and a strong convolutional baseline)

venue: In European Conference on Computer Vision, 2018

authors:
Dmitry Ulyanov
Andrea Vedaldi
Victor Lempitsky

title: Instance normalization: The missing ingredient for fast stylization

venue: arXiv preprint arXiv:1607.08022,

authors:
Catherine Wah
Steve Branson
Peter Welinder
Pietro Perona
Serge Belongie

title: The caltech-ucsd birds-200-2011 dataset

authors:
Yaming Wang
Vlad I Morariu
Larry S Davis

title: Learning a discriminative filter bank within a cnn for fine-grained recognition

authors:
Zhongdao Wang
Luming Tang
Xihui Liu
Zhuliang Yao
Shuai Yi
Jing Shao
Junjie Yan
Shengjin Wang
Hongsheng Li
Xiaogang Wang

title: Orientation invariant feature embedding and spatial temporal regularization for vehicle re-identification

authors:
Peter Welinder
Steve Branson
Takeshi Mita
Catherine Wah
Florian Schroff
Serge Belongie
Pietro Perona

title: Caltech-ucsd birds

authors:
Saining Xie
Ross Girshick
Piotr Dollár
Zhuowen Tu
Kaiming He

title: Aggregated residual transformations for deep neural networks

authors:
Hantao Yao
Shiliang Zhang
Richang Hong
Yongdong Zhang
Changsheng Xu
Qi Tian

title: Deep representation learning with part loss for person re-identification

authors:
Yue Yao
Liang Zheng
Xiaodong Yang
Milind Naphade
Tom Gedeon

title: Simulating content consistent vehicle datasets with attribute descent

venue: In European Conference on Computer Vision,

authors:
Chaojian Yu
Xinyi Zhao
Qi Zheng
Peng Zhang
Xinge You

title: Hierarchical bilinear pooling for fine-grained visual recognition

authors:
Lianbo Zhang
Shaoli Huang
Wei Liu
Dacheng Tao

title: Learning a mixture of granularity-specific experts for fine-grained categorization

venue: In IEEE International Conference on Computer Vision, 2019

authors:
Ning Zhang
Ryan Farrell
Forrest Iandola
Trevor Darrell

title: Deformable part descriptors for fine-grained recognition and attribute prediction

authors:
Ning Zhang
Jeff Donahue
Ross Girshick
Trevor Darrell

title: Part-based r-cnns for fine-grained category detection

authors:
Heliang Zheng
Jianlong Fu
Tao Mei
Jiebo Luo

title: Learning multi-attention convolutional neural network for fine-grained image recognition

authors:
Heliang Zheng
Jianlong Fu
Zheng-Jun Zha
Jiebo Luo

title: Looking for the devil in the details: Learning trilinear attention sampling network for fine-grained image recognition

authors:
Liang Zheng
Liyue Shen
Lu Tian
Shengjin Wang
Jingdong Wang
Qi Tian

title: Scalable person re-identification: A benchmark

authors:
Zhedong Zheng
Liang Zheng
Yi Yang

title: Unlabeled samples generated by gan improve the person re-identification baseline in vitro

authors:
Yi Zhou
Ling Shao

title: Viewpoint-aware attentive multi-view inference for vehicle reidentification

heading: 1 INTRODUCTION

text:
The main goal of fine-grained recognition is to discriminate the sub-categories of the images that belong to a same general class. Its practical applications include fine-grained image classification (Welinder et al., 2010; Krause et al., 2013; Maji et al., 2013), re-identification (Zheng et al., 2017b; Liu et al., 2016b;a), etc. These tasks are fundamentally challenging due to considerable inter-class similarity. Consequently, an ideal feature extractor should be capable of extracting discriminative fine-grained features from subtle image regions, such as the beak to classify the bird species or the stickers on the windshield to identify the vehicle.
While the general design of Convolutional Neural Network (CNN) (Simonyan & Zisserman, 2014; He et al., 2016) applies either strided convolution or strided pooling as downsampling operations to reduce the computational requirement of the network and also increase the receptive field of subsequent convolutions, these operations, however, have been substantiated to be harmful to fine-grained recognition (Sun et al., 2018; Luo et al., 2019a). Its disadvantage is two-fold: 1) false assumption on spatial equality which would overemphasize the background region while neglecting the smaller but more discriminative regions and 2) excessive reduction of resolution which would spoil the granularity of feature. To cope with the problem, existing approaches (Zhang et al., 2014; Fu et al., 2017; Chen & Deng, 2019; Zhang et al., 2019) introduce cropping operation that guides the network to only focus on the most critical region while preserving more details in the sampled part. However, to maintain computational complexity, full conservation of feature resolution would instead restrict a limited scale, or area, of the cropped region. Comprehensively, as shown in Figure 1(a)(c), the problems of two commonly used downsamplings can be summarized into a trade-off problem between the scale of receptive field and the granularity of representation.
To tackle the problem, in this work, we introduce a novel pooling algorithm, named Adaptive Region Pooling (ARP), to bridge strided and cropping operations. It consists of two steps. First, ARP automatically crops the feature from the most critical region with a well-estimated and adaptive cropping scale. Second, after being cropped into different sizes, the feature is further downsampled to a consistent size through bilinear downsampling. As illustrated in Figure 1(b), with the meticulous evaluation of cropping scale, ARP can better handle the trade-off issue and leverage the
beneficial properties of both strided and cropping operations. Besides, ARP also features other three advantages. First, it is incorporated with a controllable mechanism that supports users to actively reach a more satisfying balance in the trade-off problem, as in two cases in Figure 1(b). Second, compared to the previous work (Lin et al., 2015a; Huang et al., 2016; Kim et al., 2018; Zheng et al., 2019) which implement the cropping operations by an additional neural network, ARP utilizes a more efficient yet effective operation without any learning-based parameters. Hence, it makes the network achieve lower computational overhead, experience a stabler training process, and converge more quickly. Finally, ARP can be simply plugged into any CNN layer and the network can be optimized in a one-stage end-to-end manner, which is completely identical to the training of original CNN backbone.
At last, we demonstrate that ARP could facilitate fine-grained recognition through our Multiple Scale and Granularity Network (MSGN). In MSGN, ARP serves as a downsampling alternative to extract the feature of finer details complementary to the coarse feature provided by the typical downsampling method. The contributions of this paper can be summarized as follows:
• We introduce Adaptive Region Pooling (ARP): a novel downsampling that smoothly bridges strided pooling and cropping operations. The sub-sampled representation focuses on the most discriminative region and simultaneously contains more fine-grained information. • To our knowledge, ARP is the first pooling algorithm with an adjustable trade-off mechanism that allows users to apply the human domain knowledge of the target task to strike a more desirable balance between the scale of receptive field and the granularity of downsampled feature. • With the usage of ARP, MSGN outperforms the existing frameworks in terms of both effectiveness and efficiency on multiple benchmarks for both the tasks of image classification and retrieval.

heading: 2 RELATED WORK

text:
Fine-grained recognition focuses on learning discriminative representations to classify or identify the images within a same general category. There are two common applications: re-identification and fine-grained image classification.
Re-identification (re-ID) studies the problem of identifying the individuals of persons (Zheng et al., 2015; 2017b; Ristani et al., 2016) or vehicles (Liu et al., 2016b;c;a) in different camera views. To distinguish two similar targets with only slight differences, most of the existing methods (Yao et al., 2019; Kalayeh et al., 2018; Liu et al., 2021; Wang et al., 2017; Zhou & Shao, 2018; Chen et al., 2020b) adopt spatial attentive mechanism to emphasize the features extracted from critical parts, such as the head for person re-ID or the tires for vehicle re-ID. While previous work get significant improvement, they generally suffer from massive loss of fine-grained details when CNN backbone downsamples the feature through strided operations. Sun et al. (2018) and Luo et al. (2019a) both
find a huge performance gain in the task of person re-ID, numerically 1.4% for mAP on Market1501 (Zheng et al., 2015), by simply removing the pooling operation in the last CNN layer.
Fine-grained image classification (FGIC) aims to classify the sub-classes of images, such as the species of birds (Welinder et al., 2010) or the models of vehicles (Krause et al., 2013). Compared to re-ID, the datasets of FGIC have a severe problem of image misalignment which impels lots of work to apply another sampling operation, cropping, to align the image or representation and also guide the network to focus on the sampled region. Several prior work (Welinder et al., 2010; Zhang et al., 2013; 2014; Lin et al., 2015a; Huang et al., 2016; Kim et al., 2018; He et al., 2019) leverage extra object detection datasets or human-labeled bounding box annotations to learn the localization of discriminative regions. However, such auxiliary labels are hard to acquire in the real-world and large-scale scenarios. Recently, there have been emerging work with a more general setting by proposing unsupervised approaches. Jaderberg et al. (2015) introduce a differentiable STN that can learn to well align the images or representations by themselves. While it can perform well on the digit datasets (LeCun et al., 1998; Netzer et al., 2011), lack of explicit guidance leads to the unstable training process and being hard to converge on more sophisticated tasks (see Section 4.3 for more details and discussions). As the following work, Fu et al. (2017) propose RACNN which is a cascade coarse-to-fine CNN model. The similar architecture is also adopted by Chen & Deng (2019) and Zhang et al. (2019). Although they show the great capability of cropping the discriminative region, the cascade architecture requires multi-stage training for better optimization and struggles with the requirement of heavy computation. In this work, we also introduce a sampling operation with an unsupervised approach; however, Adaptive Region Pooling (ARP) contains neither additional neural network nor extra learning-based parameters and makes the network using ARP more computationefficient and able to be optimized in a simple one-stage end-to-end manner.

heading: 3 PROPOSED METHODOLOGY

text: In this section, we introduce the proposed pooling algorithm in three parts. First, in Section 3.1, Region Pooling (RP) considers a simplified task which aims to crop the hidden representation from the most critical region with a fixed cropping scale. Then, in Section 3.2, Adaptive Region Pooling (ARP) estimates the adaptive cropping scale for each representation to better capture the interesting object. Finally, we introduce Multiple Scale and Granularity Network (MSGN) as an example of network architecture with the usage of ARP in Section 3.3.

heading: 3.1 REGION POOLING

text:
Given a CNN feature and a fixed downsampling rate, the main goal of Region Pooling (RP) is to crop the feature from the most critical region into a consistent scale. An intuitive approach is to use an extra network to predict the cropping center which nevertheless, often suffers from the requirement of expensive bounding box annotations or complicated multi-stage learning strategy to train the network. In contrast, RP uses a more efficient yet effective operation with two steps.
In the first step, as shown in Figure 2 (a), RP would evaluate the significance score for each location and generate the keypoint confidence map C based on the C-dim feature vectors, where C denotes the number of channels. Here, we define “significance” in an engineering sense as follows: a location, whose feature vector includes more highly-activated neurons, is more significant. The physical meaning is that a channel-wise feature map is actually a detector of a certain semantic object (Chen et al., 2017; 2020a), such as a bird’s beak or wings. For a well-trained CNN, most of the channels would detect the objects which are more beneficial to the target task; thus, the locations activated by more channels typically cover critical semantic objects and are more significant.
Based on the definition, we then give a formal formulation as follows. For the input feature F ∈ RC×H×W with width W , height H and C channels and its element (or activated value) denoted as x(i,j,k) at ith channel, jth row and kth column, in order to balance the influence of each channel in the subsequent operations, we first standardize each channel-wise feature map as:
x̂(i,j,k) = x(i,j,k) − µ(i)
σ(i) , where µ(i) =
H∑ j W∑ k x(i,j,k)
H ·W , σ2(i) =
H∑ j W∑ k (x(i,j,k) − µ(i))2
H ·W , (1)
and x̂ represents the element of the standardized feature. Such standardization can be simply implemented by Instance Normalization (Ulyanov et al., 2016). Afterwards, we evaluate the significance score of each location based on the average of their feature vectors with the form as:
s(j,k) =
C∑ i=1 x̂(i,j,k)
C (2)
where s(j,k) represents the significance score of the location (j, k). Complied with our definition, Equation 2 assigns larger s for the locations that are highly activated in more channels. Next, we estimate the confidence of a location to be a “keypoint”, or a critical location, by Sigmoid function σ(·). Specifically, we generate a keypoint confidence map C ∈ RH×W by:
C = [ c(1,1), ..., c(H,W ) ] = [ σ(s(1,1)), ..., σ(s(H,W )) ] (3)
where c(j,k) denotes the keypoint confidence at the location (j, k). We visualize the generated keypoint confidence maps C in Figure 5 and 6. If we project them to the original images, the locations with larger keypoint confidences effectively represent the more discriminative regions, such as the beak and colorful feather of a bird or the bumper and lamp of a vehicle.
In the second step, as in Figure 2(c), given a fixed downsampling rate DSR = [DSRH , DSRW ], RP would crop the most critical region with height (H · DSRH ) and width (W · DSRW ). To this end, we apply a convolution operation on the keypoint confidence map C with a kernel K ∈ R(H·DSRH)×(W ·DSRW ). The region with the largest weighted summation of keypoint confidences would be selected as the cropped region, which can be written as:
(j∗, k∗) = arg max (j,k) (K C(j,k)) (4)
where represents inner product, C(j,k) is the cropped keypoint confidence map centered at the location (j, k), and (j∗, k∗) indicates the center location of the selected cropped region.
Discussion of kernel K. Empirically, we find that using Gaussian filter as kernel K yields the best performance. A potential explanation is that it tends to select the region with more keypoints locating near the center of cropped region and hence, also solves the problem of feature misalignment.

heading: 3.2 ADAPTIVE REGION POOLING

text:
Although Region Pooling demonstrates the great ability to sample the feature from the most critical region, nonetheless, there is no single downsampling rate suitable for all features. Specifically, the features with highly separate keypoints should have a larger downsampling rate to cover a wider interesting region while in contrast, the ones with more concentrated keypoints should use a smaller rate. To meet this requirement, we further introduce Adaptive Region Pooling (ARP) that automatically estimates a better adaptive downsampling rate based on the distribution of keypoints, or more precisely: the standard deviation (abbreviated as std) of keypoints distribution.
As depicted in Figure 2(b), to evaluate the adaptive downsampling rate of height DSRadpH or width DSRadpW , ARP respectively refers the vertical or horizontal normalized distribution of keypoints:
Dv = [ W∑ k=1 c(1,k) Σ C , ..., W∑ k=1 c(H,k) Σ C ] ∈ RH , Dh =  H∑ j=1 c(j,1) Σ C , ..., H∑ j=1 c(j,W ) Σ C  ∈ RW (5) where ΣC is the summation of keypoint confidence map.
Since the derivations of DSRadpH and DSR adp W take similar forms, we use the former as an example in the following. Here, we first consider a simplified condition that each image only contains one consecutive interesting object and the keypoints uniformly distribute in the target object. We will explore more circumstances in the following discussion and Appendix A. Under this assumption, the keypoints distribution can be regarded as a discrete uniform distribution:
Dv(y) =  1b− a , a < y 6 b0 , otherwise , ∀ 0 ≤ a < b ≤ H, (6) with its std:
σ(Dv) = √
(b− a)2 − 1 12 ≈ b− a√ 12 . (7)
Note that Dv has the keypoints all lying within the interval (a, b]. Therefore, if the cropped region is expected to cover all keypoints while ignoring the background, the ideal length of cropped region should be (b− a) and hence, the expected downsampling rate should be (b− a)/H . Combing with Equation 7, we can rewrite the adaptive downsampling rate as:
DSRadpH = b− a H =
√ 12
H · σ(Dv). (8)
Finally, with Equation 8, we can use the std of keypoints distribution to well estimate the adaptive downsampling rate that could exactly include all keypoints in the cropped region.
Discussion of different distributions D. Intuitively, in Equation 8, the adaptive downsampling rate is proportional to the std of keypoints distribution. Therefore, for more sparse or dense keypoints distribution (e.g., M-shaped or bell-shaped distribution), the cropped region could automatically be wider or narrower to better capture the interesting object. In Appendix A, we also empirically test the stability and rationality of Equation 8 on various randomly simulated distributions and show that Equation 8 can consistently evaluate an intelligible adaptive downsampling rate that can exactly cover most of the keypoints.
Subsequently, as in Figure 2, ARP first crops the feature F from the most critical region based on Equation 4 and adaptive downsampling rate as Fcrop ∈ RC×(H·DSR adp H )×(W ·DSR adp W ) and then, subsamples it to a consistent size as F′ ∈ RC×(H·DSRH)×(W ·DSRW ) through bilinear downsampling. Such a “cropping and downsampling” process can be viewed as the trade-off between the scale of receptive field and the resolution of feature. Specifically, the feature with a larger cropped region would encounter a huge reduction of the feature granularity in the following downsampling which makes ARP act as strided operation. In contrast, a smaller cropped region would preserve more finegrained details in the downsampled feature; in such a case, ARP intrinsically behaves like cropping function. To allow users to strike a better balance in the trade-off problem, we further incorporate ARP with a controllable mechanism by revising Equation 8 as:
DSRadpH = min
( max ( k · √ 12
H · σ(Dv), DSRH
) , 1 ) . (9)
k is the trade-off coefficient and min-max operation ensures a reasonable downsampling rate.
Discussion of trade-off coefficient k. k is a meaningful coefficient. When it is set to 1, the cropped region would nearly cover the whole target object, such as an entire bird or vehicle. And if a smaller k is set, the length of cropped region would be proportionally decreased. Through k, users can actively reach a satisfying sampling scale according to the human domain knowledge of the target task. Take the below case of Figure 1(b) as an example, users can set a smaller k to extract the feature only from the front face of vehicle, which usually contains many identity-relative features, rather than the whole vehicle. Hence, the sub-sampled feature would keep richer fine-grained details.

heading: 3.3 MULTIPLE SCALE AND GRANULARITY NETWORK

text: We introduce Multiple Scale and Granularity Network (MSGN) as an example of the network architecture that uses ARP as the downsampling operations. As in Figure 3, MSGN has one low-level feature extractor followed by multiple parallel high-level extractors. Considering that the hidden representations before Conv3 merely contain the information of low-level patterns which are unreliable to determine the critical region, we only adopt ARP with k = 1 in Conv3 to remove some background regions in the early stage. As for the following high-level extractors, including one global and several local view branches, we respectively apply strided operation and ARP with smaller k in Conv4 to downsample the feature with different scales and granularities. We also follow the strategy from Sun et al. (2018) and Luo et al. (2019a) to remove pooling operations in Conv5 of local view branches. Finally, the features extracted from the multiple branches would be concatenated and fed into a fully connected layer to comprehensively generate the final representative embedding by referring to the features from both global and local views. We will first validate MSGN with one local view branch in Section 4 and further, discuss the network with more branches in Appendix B.

heading: 4 EXPERIMENTS

text: In the following, we evaluate ARP through MSGN in both tasks of fine-grained image classification (FGIC) and vehicle re-identification (re-ID) for image retrieval.

heading: 4.1 IMPLEMENTATION DETAILS

text: We implement MSGN with three backbones: ResNet-50, ResNet-101 and ResNeXt-101 32x8d (He et al., 2016; Xie et al., 2017). For a fair comparison, we set DSR of ARP as [0.5, 0.5] to share the same height and width of the downsampled feature as from the pooling with stride = 2. Following the same protocol by Zhang et al. (2019) and Ji et al. (2020), we augment input images by resizing to 512× 512 then randomly cropping to 448× 448 with random horizontal flipping and use crossentropy to supervise the training. For vehicle re-ID, as in standard literature (Luo et al., 2019a; Chen et al., 2020b), we directly resize images to 224× 224 and use the combinations of batch-hard triplet loss (Hermans et al., 2017) and cross-entropy loss for the supervision. The training lasts for 100 epochs. More training and testing details can be found in Appendix C.

heading: 4.2 COMPARISON WITH THE STATE-OF-THE-ARTS

text:
Fine-grained image classification. We first evaluate MSGN on FGIC in terms of both effectiveness and efficiency and report the results in Table 1. MSGN achieves the new state-of-the-art performance: 90.2% on CUB-200-2011, 95.6% on Stanford Cars, and 94.4% on FGVC-Aircraft with ResNeXt-101. With ResNet-50 or ResNet-101, MSGN also shows a consistent improvement over the previous counterparts with the same CNN backbone. In addition to effectiveness, our model also achieves a significant enhancement in efficiency. While previous work apply the additional network
to generate the sub-sampling parameters (Lin et al., 2015a; Zheng et al., 2019; Ding et al., 2019) or implement the cascade coarse-to-fine architecture (Fu et al., 2017; Chen & Deng, 2019; Zhang et al., 2019), ARP only utilizes efficient operations for sampling. Therefore, the computational cost of MSGN is approximately half of the previous models, specifically 44.5% lower than CrossX (Luo et al., 2019b) on ResNet-50 and 52.7% lower than MGE (Zhang et al., 2019) on ResNet-101.
Vehicle re-identification. We also evaluate the representation learning of MSGN on vehicle reID and report the results in Table 2. Prior approaches (Wang et al., 2017; Zhou & Shao, 2018; Meng et al., 2020; Chen et al., 2020b) mostly focus on spatial attentive mechanism but ignoring the loss of fine-grained information when the representation are sub-sampled through strided operation. In contrast, MSGN effectively solves the problem by using ARP as the downsampling operation. Hence, MSGN outperforms the previous frameworks on both VeRi-776 and VehicleID, especially on VeRi-776 by a notable margin of 2.2% on mAP compared to SAVER (Khorramshahi et al., 2020).

heading: 4.3 COMPARISON WITH SPATIAL TRANSFORMER NETWORK

text:
Spatial Transformer (ST) (Jaderberg et al., 2015) module is a differentiable sampling operation that can automatically learn to perform an affine transformation on an image or a hidden representation. In light of the similar functionality between ST module and ARP, we additionally compare these two poolings. In the experiment, we adopt ST module or ARP as the downsampling operation in Conv4 of local view branch in MSGN and plot the learning curves and the learned sampling outcomes in Figure 4. We then discuss the comparison under two aspects: stability and intervenability.
First, as shown in Figure 4(a), the network using ST module comparably experiences a slower learning progress, reaches later convergence, and eventually performs worse at the end of training. We can find a possible explanation from Figure 4(b); since ST module relies on learning-based parameters for sampling, in the first ten epochs, it performs unstably and generates unaccountable downsampling results which instead disturb the training. In contrast, ARP is able to correctly sample from the foreground object (red box in Figure 4(b)) even in the first epoch. Following the training, with more channels detecting the beneficial vehicle parts, the sampled region progressively covers the entire front face of vehicle which contains more identity-relative features. It is also worth noting that due to the instability of ST module, in the paper (Jaderberg et al., 2015), ST module is restricted to perform the 2 DoF transformation (i.e., only translation) in the harder task, like FGIC.
Next, at the end of the training, ST module fails to learn an effective transformation which is supposed to concentrate on the more critical region. It leads to the subsequent discussion: for learningbased cropping operations, users can only passively accept the training results and lack interpretable parameters to adjust the sampling operation to better fit the target task. Conversely, owing to a meaningful trade-off mechanism, ARP allows users to actively use the prior knowledge to intervene in the representation learning and reach the more desirable sub-sampling characteristic.

heading: 4.4 ABLATION STUDIES

text: We evaluate the effectiveness of each component in the proposed framework and report the results in Table 3. The baseline model simply uses the original ResNet-50 as the extractor. We then replace the downsampling in Conv4 with ARP of k = 0.9. Leveraging the attention on more discriminative regions, the model achieves the improvement of 0.6% on CUB-200-2011 and 0.5% on VeRi-776. Next, as listed in the third to fifth rows in Table 3, we apply another high-level extractor with three different downsampling operations in Conv4, including strided convolution, RP with DSR = [0.5, 0.5], and ARP with k = 0.7. It shows that, with a parallel branch, the performances are all boosted. However, without using RP and ARP, the network comparably performs worse due to the lack of fine-grained information. Furthermore, the network with ARP yields the best performance, which describes that ARP can adaptively estimate the better downsampling rate for each representation and properly cover the critical region. Finally, with all proposed components, MSGN with ARP outperforms the baseline model by 3.5% on CUB-200-2011 and 2.3% on VeRi-776.

heading: 4.5 VISUALIZATION

text:
Qualitative results of Adaptive Region Pooling. To understand the practical effect of ARP, we plot the keypoint confidence maps C and the cropped regions in Figure 5 and 6 (especially for the final setting of k = 0.7). Note that the demonstrated images are in various manners, such as different actions of birds (e.g., standing and flying) and different viewpoints or types of vehicles (e.g., truck, bus, SUV, and sedan) to validate the robustness of ARP. First, as in Figures 5(b) and 6, the keypoint confidence maps C from Conv4 are able to recognize the critical locations in images, such as the bird’s beak and colorful feathers or the vehicle’s bumper and decoration. Along with the proper adaptive downsampling rate, the sampled regions can cover the most critical and consistent part in different images. Also, as in Figure 5(a), ARP in Conv3 can eliminate some negligible background regions in the early stage while enriching the feature resolution in the foreground object.
Selection of trade-off coefficient k. We visualize the cropped regions and list the validation results with different k in Figure 6. As in the second to fourth columns, users can set k = 0.5 or 0.7 or 0.9 if the sampled region is respectively expected to cover merely a part of front face, or the whole front face with the windshield, or nearly the entire vehicle. Commonly, the front face of vehicle includes several important but fine identity-relative features which are beneficial to vehicle re-ID. Hence, with the aid of domain knowledge, the final setting of k = 0.7 yields the best performance.

heading: 5 CONCLUSION

text: In this paper, we propose a novel downsampling operation, Adaptive Region Pooling (ARP), which automatically samples the feature from the most critical region and in the same time, increases the feature granularity. Besides, ARP owns an adjustable trade-off mechanism which supports users to actively balance the scale of receptive field and the resolution of sub-sampled feature. Without any learning-based parameters, ARP can be simply integrated into a CNN backbone and the network can be stably end-to-end optimized. The experiments show the effectiveness and efficiency of the proposed model in both the tasks of image classification and image retrieval.

heading: A ROBUSTNESS TESTING OF ADAPTIVE DOWNSAMPLING RATE

text:
In Equation 8, we show that the adaptive downsampling rate of height or width can be respectively derived by the standard deviation (abbreviated as std) of 1-D vertical or horizontal keypoints distribution. In this section, we further evaluate the stability and rationality of Equation 8 on the various simulated keypoints distributions. We comply with the following rules to simulate the distributions: 1) most of the keypoints locate in one consecutive interval; 2) the interval can have any length and can be anywhere in the entire distribution; 3) the distribution within the interval is random. These rules reflect the fact that the images for fine-grained recognition typically contain only one target object (e.g., a bird or a vehicle) with various scales and locations, and the critical features (e.g., a bird’s beak or a vehicle’s tires) are randomly distributed in the target object. Following these principles, we simulate the distributions by first assigning a random significance score at each location and filtering by a 1-D Gaussian kernel with a random mean and std. We plot the distributions and also show the cropped lengths and regions in Figure 7.
We can observe that for each keypoints distribution with different sparsity or at a different location, Equation 8 can consistently evaluate an appropriate adaptive downsampling rate that can cover most of the keypoints while removing the negligible background region with a relatively few keypoints. Subsequently, through the trade-off mechanism, users can actively adjust the scale of cropped region to better capture the interesting parts in the target object.

heading: B MULTIPLE SCALE AND GRANULARITY NETWORK WITH MORE LOCAL VIEW BRANCHES

text:
In Section 4, we have shown the effectiveness of MSGN with one local view branch (LVB) and will further discuss the behavior of MSGN with more (i.e. 2 and 4) LVBs in the following. To avoid excessive overlaps among the sampled regions from different LVBs, we set a smaller k = 0.5 for Adaptive Region Pooling (ARP) in Conv4 to reduce the scale of cropped region. We evaluate the models on two datasets: CUB-200-2011 (Wah et al., 2011) and VeRi-776 (Liu et al., 2016b;c), and respectively report the visualization results and the validation performance in Figure 8 and Table 4.
We observe that, for MSGN with two LVBs, two branches can learn to focus on the scattered parts in the target object, such as the head and torso of a bird, and can jointly cover larger critical regions
in contrast to the one-branch model (by the comparison with the visualization in Figure 5). Also, due to smaller k, the model can leverage the higher resolution of sub-sampled feature and the richer details in the cropped region. Therefore, as listed in Table 4, we can get a performance gain when we increase the number of LVB from one to two. However, there is no more improvement when we further apply four LVBs. A possible explanation is that the sampled regions from the two-branches model have already covered most of the parts in the interesting object; therefore, the extra LVBs would merely focus on the overlapping regions and provide redundant information.

heading: C TRAINING AND VALIDATING DETAILS

text: The network is initialized with ImageNet pretrained weights. During the training phase of 100 epoch, we set the initial learning rate as 0.001 and adopt a linear warm-up strategy to linearly increase the learning rate to 0.01 in the first ten epochs; then, we use a multi-stage learning rate scheduler by multiplying the learning rate by 0.1 in 40th and 70th epoch. We use SGD optimizer with a weight decay of 0.0005 and momentum of 0.9. Considering that the images for vehicle reidentification are usually well-aligned, hence, we use original strided pooling as downsampling in Conv3. As for fine-grained image classification, during the inference phase, the input images are resized to 512×512 then center-cropped to 448×448. Our network is implemented in PyTorch and all experiments have been conducted on one NVIDIA Titan RTX GPU with 24GB memory.