summaries: The authors present a method to jointly learn a shape and UV parameterization of a creased document observed from multiple views. They use an SDF based implicit shape representation in combination with a recent neural volumetric renderer IDR, and optimize the energy composed of the main photometric term (a discrepancy between the observed and rendered image) and regularization terms enforcing well behaved UV space and a fixed scale conformal mapping which corresponds to a behaviour of a real paper material. Since the implicit field representation does not explicitly model a surface, the authors cannot directly optimize the 3D->2D mapping using standard graphics techniques and rather adapt the IDR renderer to be conditioned on the locations sampled from the UV space and train it jointly with the MLPs defined for the bijective mapping 2D->3D and 3D->2D. The authors show that they can get better results both quantitatively and qualitatively when compared to one of the SotA works DewarpNet. **High-Level Overview**  This paper proposes a neural rendering technique to learn how to estimate an implicit surface and its UV-parameterization from a set of 2D images. The proposed approach is applied to the problem of document unwarping, wherein a wrinkled, folded, or bent document (e.g. a piece of paper with text on it) can be digitally flattened with its contents preserved. For good measure, the paper also shows that learning the texture mapping enables the contents of the unwarped document to be edited in the texture space. The proposed approach is evaluated against a prior work, DewarpNet (Das et al., 2019) on the basis of Multi-Scale Structural Similarity (MS-SSIM) and Local Distortion (LD) as well as the auxiliary task of OCR.  **Key Contributions**  * Builds on IDR (Yariv et al., 2020) to learn an explicit bijective texture mapping for a learned implicit surface * A loss function formulation for the framework * Application of learned implicit surfaces for document unwarping with reduction in texture distortion and improvements in subsequent OCR metrics  **How It Works**  * Starts with the same problem formulation as IDR: an MLP to model the surface (Eq. 1) and an appearance MLP to model the color (Eq. 2) * Modifies the appearance MLP to also be a function of texture coordinates (Eq. 3) * Adds an additional forward and backward MLP to learn the bijective UV mapping * Jointly trains all MLPs using a pixelwise rendering loss This paper presents document unwrapping based on optimization of neural networks. Four MLPs are defined to model the implicit surface of the target, the colors of the target, the mapping function from 3D points to 2D texture coordinates, and the inverse mapping from 2D to 3D. Similarly to NERF, the target example is approximated by training the MLPs using differentiable rendering.    The paper presents a multi-view method for undistorting documents. The method makes use of an existing method (IDR) for reconstructing distorted documents in 3-D, with an additional bijective neural network to parameterize distorted surfaces by rectangles. The method significantly outperforms SOTAs in the experiments.  
abstractText: In this paper, we present a novel approach to learn texture mapping for a 3D surface and apply it to document image unwarping. We propose an efficient method to learn surface parameterization by learning a continuous bijective mapping between 3D surface positions and 2D texture-space coordinates. Our surface parameterization network can be conveniently plugged into a differentiable rendering pipeline and trained using multi-view images and rendering loss. Recent work on differentiable rendering techniques for implicit surfaces has shown high-quality 3D scene reconstruction and view synthesis results. However, these methods typically learn the appearance color as a function of the surface points and lack explicit surface parameterization. Thus they do not allow texture map extraction or texture editing. By introducing explicit surface parameterization and learning with a recent differentiable renderer for implicit surfaces, we demonstrate state-of-the-art document-unwarping via texture extraction. We show that our approach can reconstruct high-frequency textures for arbitrary document shapes in both synthetic and real scenarios. We also demonstrate the usefulness of our system by applying it to document texture editing.
references.authors: David Bau Hendrik Strobelt William Peebles Jonas Wulff Bolei Zhou Jun-Yan Zhu Antonio Torralba
references.title: Semantic photo manipulation with a generative image prior
references.venue: ACM Transactions on Graphics (TOG),
references.authors: Jan Bednarik Shaifali Parashar Erhan Gundogdu Mathieu Salzmann Pascal Fua
references.title: Shape reconstruction by learning differentiable surface representations
references.venue: In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition,
references.authors: Alexander W. Bergman Petr Kellnhofer Gordon Wetzstein
references.title: Fast training of neural lumigraph representations using meta learning, 2021
references.authors: Sai Bi Nima Khademi Kalantari Ravi Ramamoorthi
references.title: Patch-based optimization for image-based texture mapping
references.venue: ACM Transactions on Graphics (TOG),
references.authors: Caroline Chan Shiry Ginosar Tinghui Zhou Alexei A Efros
references.title: Everybody dance now
references.venue: In Proceedings of the International Conference on Computer Vision,
references.authors: Tony Chan Wei Zhu
references.title: Level set based shape prior segmentation
references.venue: In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition
references.authors: Anpei Chen Zhang Chen Guli Zhang Kenny Mitchell Jingyi Yu
references.title: Photo-realistic facial details synthesis from single image
references.venue: In Proceedings of the International Conference on Computer Vision,
references.authors: Anpei Chen Zexiang Xu Fuqiang Zhao Xiaoshuai Zhang Fanbo Xiang Jingyi Yu Hao Su
references.title: MVSNeRF: Fast generalizable radiance field reconstruction from multi-view stereo
references.venue: arXiv preprint arXiv:2103.15595,
references.authors: Zhiqin Chen Hao Zhang
references.title: Learning implicit fields for generative shape modeling
references.venue: In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,
references.authors: Christopher B Choy Danfei Xu JunYoung Gwak Kevin Chen Silvio Savarese
references.title: 3d-r2n2: A unified approach for single and multi-view 3d object reconstruction
references.venue: In European conference on computer vision,
references.authors: Sagnik Das Gaurav Mishra Akshay Sudharshana Roy Shilkrot
references.title: The Common Fold: Utilizing the Four-Fold to Dewarp Printed Documents from a Single Image
references.venue: In Proceedings of the 2017 ACM Symposium on Document Engineering,
references.authors: Sagnik Das Ke Ma Zhixin Shu Dimitris Samaras Roy Shilkrot
references.title: DewarpNet: Single-image document unwarping with stacked 3D and 2D regression networks
references.venue: In Proceedings of the International Conference on Computer Vision,
references.authors: Sagnik Das Kunwar Yashraj Singh Jon Wu Erhan Bas Vijay Mahadevan Rahul Bhotika Dimitris Samaras
references.title: End-to-end piece-wise unwarping of document images
references.venue: In Proceedings of the IEEE/CVF International Conference on Computer Vision,
references.authors: Jiankang Deng Shiyang Cheng Niannan Xue Yuxiang Zhou Stefanos Zafeiriou
references.title: Uv-gan: Adversarial facial uv map completion for pose-invariant face recognition
references.venue: In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition,
references.authors: Stephan J. Garbin Marek Kowalski Matthew Johnson Jamie Shotton Julien Valentin
references.title: Fastnerf: High-fidelity neural rendering at 200fps, 2021
references.authors: Amos Gropp Lior Yariv Niv Haim Matan Atzmon Yaron Lipman
references.title: Implicit geometric regularization for learning shapes
references.venue: arXiv preprint arXiv:2002.10099,
references.authors: Thibault Groueix Matthew Fisher Vladimir G Kim Bryan C Russell Mathieu Aubry
references.title: A papier-mâché approach to learning 3d surface generation
references.venue: In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition,
references.authors: Steven Haker Sigurd Angenent Allen Tannenbaum Ron Kikinis Guillermo Sapiro Michael Halle
references.title: Conformal surface parameterization for texture mapping
references.venue: IEEE Transactions on Visualization and Computer Graphics,
references.authors: John C Hart
references.title: Sphere tracing: A geometric method for the antialiased ray tracing of implicit surfaces
references.venue: The Visual Computer,
references.authors: Peter Hedman Julien Philip True Price Jan-Michael Frahm George Drettakis Gabriel Brostow
references.title: Deep blending for free-viewpoint image-based rendering
references.venue: ACM Transactions on Graphics (TOG),
references.authors: Hiroharu Kato Yoshitaka Ushiku Tatsuya Harada
references.title: Neural 3d mesh renderer
references.venue: In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition,
references.authors: Taeho Kil Wonkyo Seo Hyung Il Koo Nam Ik Cho
references.title: Robust Document Image Dewarping Method Using Text-Lines and Line Segments
references.venue: In Proceedings of the International Conference on Document Analysis and Recognition,
references.authors: Hyung Il Koo Jinho Kim Nam Ik Cho
references.title: Composition of a dewarped and enhanced document image from two view images
references.venue: IEEE Transactions on Image Processing,
references.authors: Tzu-Mao Li Miika Aittala Frédo Durand Jaakko Lehtinen
references.title: Differentiable monte carlo ray tracing through edge sampling
references.venue: ACM Transactions on Graphics (TOG),
references.authors: Xiaoyu Li Bo Zhang Jing Liao Pedro V. Sander
references.title: Document Rectification and Illumination Correction using a Patch-based CNN
references.venue: ACM Transactions on Graphics (TOG),
references.authors: Zhengqi Li Simon Niklaus Noah Snavely Oliver Wang
references.title: Neural scene flow fields for space-time view synthesis of dynamic scenes
references.venue: arXiv preprint arXiv:2011.13084,
references.authors: Jian Liang Daniel DeMenthon David Doermann
references.title: Geometric rectification of camera-captured document images
references.venue: IEEE Transactions on Pattern Analysis and Machine Intelligence,
references.authors: Ce Liu Jenny Yuen Antonio Torralba
references.title: Sift flow: Dense correspondence across scenes and its applications
references.venue: IEEE Transactions on Pattern Analysis and Machine Intelligence,
references.authors: Shichen Liu Tianye Li Weikai Chen Hao Li
references.title: Soft rasterizer: A differentiable renderer for image-based 3d reasoning
references.venue: In Proceedings of the International Conference on Computer Vision,
references.authors: Shichen Liu Shunsuke Saito Weikai Chen Hao Li
references.title: Learning to infer implicit surfaces without 3d supervision
references.venue: arXiv preprint arXiv:1911.00767,
references.authors: Ke Ma Zhixin Shu Xue Bai Jue Wang Dimitris Samaras
references.title: DocUNet: Document Image Unwarping via A Stacked U-Net
references.venue: In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition,
references.authors: Amir Markovitz Inbal Lavi Or Perel Shai Mazor Roee Litman
references.title: Can you read me now? Content aware rectification using angle supervision
references.venue: In Proceedings of the European Conference on Computer Vision. Springer,
references.authors: Abhimitra Meka Christian Haene Rohit Pandey Michael Zollhöfer Sean Fanello Graham Fyffe Adarsh Kowdle Xueming Yu Jay Busch Jason Dourgarian
references.title: Deep reflectance fields: Highquality facial reflectance field inference from color gradient illumination
references.venue: ACM Transactions on Graphics (TOG),
references.authors: Gaofeng Meng Zuming Huang Yonghong Song Shiming Xiang Chunhong Pan
references.title: Extraction of virtual baselines from distorted document images using curvilinear projection
references.venue: In Proceedings of the International Conference on Computer Vision,
references.authors: Gaofeng Meng Yuanqi Su Ying Wu Shiming Xiang Chunhong Pan
references.title: Exploiting Vector Fields for Geometric Rectification of Distorted Document Images
references.venue: In Proceedings of the European Conference on Computer Vision,
references.authors: Lars Mescheder Michael Oechsle Michael Niemeyer Sebastian Nowozin Andreas Geiger
references.title: Occupancy networks: Learning 3d reconstruction in function space
references.venue: In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,
references.authors: Ben Mildenhall Pratul P Srinivasan Matthew Tancik Jonathan T Barron Ravi Ramamoorthi Ren Ng
references.title: NeRF: Representing scenes as neural radiance fields for view synthesis
references.venue: In Proceedings of the European Conference on Computer Vision,
references.authors: Frederic P. Miller Agnes F. Vandome John McBrewster
references.title: Levenshtein Distance: Information Theory, Computer Science, String (Computer Science), String Metric, Damerau?Levenshtein Distance, Spell Checker
references.authors: Aymen Mir Thiemo Alldieck Gerard Pons-Moll
references.title: Learning to transfer texture from clothing images to 3d humans
references.venue: In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition,
references.authors: Michael Niemeyer Lars Mescheder Michael Oechsle Andreas Geiger
references.title: Differentiable volumetric rendering: Learning implicit 3d representations without 3d supervision
references.venue: In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,
references.authors: Jeong Joon Park Peter Florence Julian Straub Richard Newcombe Steven Lovegrove
references.title: Deepsdf: Learning continuous signed distance functions for shape representation
references.venue: In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,
references.authors: Taesung Park Ming-Yu Liu Ting-Chun Wang Jun-Yan Zhu
references.title: Semantic image synthesis with spatially-adaptive normalization
references.venue: In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition,
references.authors: Chaitanya Patel Zhouyingcheng Liao Gerard Pons-Moll
references.title: Tailornet: Predicting clothing in 3d as a function of human pose, shape and garment style
references.venue: In IEEE Conference on Computer Vision and Pattern Recognition (CVPR)
references.authors: Albert Pumarola Antonio Agudo Lorenzo Porzi Alberto Sanfeliu Vincent Lepetit Francesc Moreno-Noguer
references.title: Geometry-Aware Network for Non-Rigid Shape Prediction from a Single View
references.venue: In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition,
references.authors: Albert Pumarola Enric Corona Gerard Pons-Moll Francesc Moreno-Noguer
references.title: D-nerf: Neural radiance fields for dynamic scenes
references.venue: arXiv preprint arXiv:2011.13961,
references.authors: Johannes Lutz Schönberger Jan-Michael Frahm
references.title: Structure-from-motion revisited
references.venue: In Conference on Computer Vision and Pattern Recognition (CVPR),
references.authors: Katja Schwarz Yiyi Liao Michael Niemeyer Andreas Geiger. Graf
references.title: Generative radiance fields for 3d-aware image synthesis
references.venue: In Advances in Neural Information Processing Systems,
references.authors: Vincent Sitzmann Justus Thies Felix Heide Matthias Nießner Gordon Wetzstein Michael Zollhofer
references.title: Deepvoxels: Learning persistent 3d feature embeddings
references.venue: In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition,
references.authors: Vincent Sitzmann Michael Zollhöfer Gordon Wetzstein
references.title: Scene representation networks: Continuous 3d-structure-aware neural scene representations
references.venue: arXiv preprint arXiv:1906.01618,
references.authors: Vincent Sitzmann Julien N.P. Martel Alexander W. Bergman David B. Lindell Gordon Wetzstein
references.title: Implicit neural representations with periodic activation functions
references.venue: In arXiv,
references.authors: Chengzhou Tang Ping Tan
references.title: Ba-net: Dense bundle adjustment network
references.venue: arXiv preprint arXiv:1806.04807,
references.authors: Yuandong Tian Srinivasa G Narasimhan
references.title: Rectification and 3D reconstruction of curved document images
references.venue: In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition,
references.authors: Yochay Tzur Ayellet Tal
references.title: FlexiStickers: Photogrammetric texture mapping using casual images
references.venue: In Proceedings of the ACM SIGGRAPH Conference on Computer Graphics,
references.authors: Adrian Ulges Christoph H. Lampert Thomas Breuel
references.title: Document Capture Using Stereo Vision
references.venue: In Proceedings of the 2004 ACM Symposium on Document Engineering,
references.authors: Toshikazu Wada Hiroyuki Ukida Takashi Matsuyama
references.title: Shape from shading with interreflections under a proximal light source: Distortion-free copying of an unfolded book
references.venue: International Journal of Computer Vision,
references.authors: Nanyang Wang Yinda Zhang Zhuwen Li Yanwei Fu Wei Liu Yu-Gang Jiang
references.title: Pixel2mesh: Generating 3d mesh models from single rgb images
references.venue: In Proceedings of the European Conference on Computer Vision (ECCV),
references.authors: Zhou Wang Eero P Simoncelli Alan C Bovik
references.title: Multiscale structural similarity for image quality assessment
references.venue: In The Thirty-Seventh Asilomar Conference on Signals, Systems and Computers,
references.authors: Shih-En Wei Jason Saragih Tomas Simon Adam W Harley Stephen Lombardi Michal Perdoch Alexander Hypes Dawei Wang Hernan Badino Yaser Sheikh
references.title: Vr facial animation via multiview image translation
references.venue: ACM Transactions on Graphics (TOG),
references.authors: Fanbo Xiang Zexiang Xu Miloš Hašan Yannick Hold-Geoffroy Kalyan Sunkavalli Hao Su
references.title: NeuTex: Neural texture mapping for volumetric neural rendering
references.venue: arXiv preprint arXiv:2103.00762,
references.authors: Zexiang Xu Kalyan Sunkavalli Sunil Hadap Ravi Ramamoorthi
references.title: Deep image-based relighting from optimal sparse samples
references.venue: ACM Transactions on Graphics (TOG),
references.authors: Lior Yariv Yoni Kasten Dror Moran Meirav Galun Matan Atzmon Basri Ronen Yaron Lipman
references.title: Multiview neural surface reconstruction by disentangling geometry and appearance
references.venue: Advances in Neural Information Processing Systems,
references.authors: Shaodi You Yasuyuki Matsushita Sudipta Sinha Yusuke Bou Katsushi Ikeuchi
references.title: Multiview Rectification of Folded Documents
references.venue: IEEE Transactions on Pattern Analysis and Machine Intelligence,
references.authors: Alex Yu Vickie Ye Matthew Tancik Angjoo Kanazawa
references.title: pixelNeRF: Neural radiance fields from one or few images
references.venue: arXiv preprint arXiv:2012.02190,
references.authors: Fang Zhao Shengcai Liao Kaihao Zhang Ling Shao
references.title: Human parsing based texture transfer from single image to 3D human via cross-view consistency
references.venue: In Advances in Neural Information Processing Systems,
sections.heading: 1 INTRODUCTION
sections.text: Reconstructing 3D shapes from images is a core problem in computer vision and graphics research. With the progress in differentiable rendering (Sitzmann et al., 2019b; Kato et al., 2018; Niemeyer et al., 2020; Li et al., 2018; Liu et al., 2019b), recent learning-based 3D reconstruction approaches have achieved impressive results using 2D supervision from single image (Chen & Zhang, 2019; Groueix et al., 2018; Choy et al., 2016; Mescheder et al., 2019; Wang et al., 2018) or multi-view images (Tang & Tan, 2018; Yariv et al., 2020). These methods achieve high quality 3D reconstruction using differentiable rendering with various 3D representations such as 3D mesh (Wang et al., 2018), volumetric representation (Mildenhall et al., 2020), or implicit functions (Mescheder et al., 2019). In recent neural rendering methods such as NeRF (Mildenhall et al., 2020) and IDR (Yariv et al., 2020), continuous representations such as volume or implicit functions achieve significantly better reconstruction results than meshes or voxels because they do not discretize the 3D surface a priori. However, these continuous representations usually do not encode explicit surface parameterization, allowing 3D shape re-texturing, editing the existing texture in the 2D texture space, or recovering 2D texture from 3D surfaces. One of the most direct applications of 2D texture recovery in a geometrically constrained manner, is document unwarping, i.e., inference of a document’s flatbed-scanned version from a casual photo of a potentially creased document. Whereas 2D texture recovery could be equally valuable for other domains such as garments, or faces, the existing datasets are not directly applicable to our method.
  Our novel texture mapping approach learns surface parameterization for document unwarping by learning continuous bijective functions between 3D surface positions and 2D texture-space coordinates. We use a signed distance function (SDF) (Chan & Zhu, 2005) to represent geometry and model the appearance as a function of the 2D texture coordinates. By utilizing implicit differentiable rendering (IDR), (Yariv et al., 2020) we can reconstruct 3D shape and learn the corresponding UV parameterization of the surface simultaneously using a per-pixel rendering loss and appropriate geometric regularizations.
  We utilize two fully connected multi-layer perceptrons (MLPs) to learn a bijective mapping between 3D shapes and 2D texture space. More specifically, the forward MLP maps the 3D surface
  coordinates to 2D texture coordinates and the backward MLP maps the 2D texture coordinates to corresponding 3D surface coordinates. Following IDR (Yariv et al., 2020), we obtain the 3D surface coordinates by sphere-tracing along the ray, cast through each pixel. Our appearance rendering is formulated as a function of the 3D and the texture coordinates. Therefore, the forward and backward MLPs can be trained with a 2D pixel-wise loss between the rendered image and the given ground truth image. To the best of our knowledge, this is the first neural rendering method that can learn effective UV parameterization for implicit surfaces.
  As a corollary, our method is also the first method which utilizes implicit surface based neural rendering for document unwarping. It is a challenging task due to the presence of geometric and photometric distortions in the document. For this particular problem we introduce a prior for shapespecific texture mapping to initialize the forward MLP (3D to 2D mapping). This prior is learned from a large dataset of UV mapped document meshes, assuming that document texture space maps to a 2D equiangular quadrilateral. This assumption regularizes the forward MLP to output a highquality texture space that avoids degenerate solutions (see Fig. 3). Moreover, we introduce a conformality constraint in the backward MLP, which is consistent with how paper folds happen in the physical world, i.e., without any stretch or tear. This constraint also ensures that the backward function is bijective and smooth (Petrini et al., 2018).
  The main contributions of our paper are the following: First, we propose an efficient way to learn texture parameterization for implicit neural representations using a differentiable rendering framework. Without 3D supervision, it only requires multi-view images as ground-truth and a texture mapping prior. Second, we show that our method can be effectively used for document unwarping tasks by learning a prior for explicit texture mapping on the document shape. We show that this prior can be learned from a dataset of texture-mapped meshes. Third, we show that our method is effective for document image unwarping and texture editing (see Fig. 1). We achieve a 52% relative improvement over the publicly available state-of-the-art1 (Das et al., 2019) in terms of mean local distortion across 500 views from ten synthetic scenes. Additionally, we achieve a ∼25% improvement in optical character recognition (OCR) in terms of character and word error rate.
sections.heading: 2 PREVIOUS WORK
sections.text: Neural Rendering. Neural rendering generates images and videos by integrating conventional computer graphics rendering pipelines into deep neural networks (Tewari et al., 2020). It enables explicit or implicit control of scene properties, including illumination, geometry, texture, etc. Neural rendering can synthesize semantic photos (Park et al., 2019b; Bau et al., 2019), novel views (Hedman et al., 2018; Sitzmann et al., 2019a), relighting (Xu et al., 2018; Meka et al., 2019), facial/body reenactment (Chan et al., 2019; Wei et al., 2019), estimate scene properties etc. Kato (Kato et al., 2018) proposed a differentiable neural renderer using an approximate gradient for rasterization. Liu (Liu et al., 2019a) proposed SoftRas, which extended differentiable rasterization. Li (Li et al., 2018) further demonstrated the feasibility of integrating ray-tracing in deep neural networks. More recently, implicit surface or volume rendering has become mainstream in neural rendering approaches such as IDR (Yariv et al., 2020) and NeRF (Mildenhall et al., 2020). These approaches are based
  1A more recent approach, CREASE (Markovitz et al., 2020), data and models are not publicly available.
  on multi-view surface reconstruction to associate the scene geometry to the appearance in different views. NeRF is extended to lot of variants including PixelNeRF (Yu et al., 2020), MVSNeRF (Chen et al., 2021), dynamic NeRF (Li et al., 2020; Pumarola et al., 2020), GRAF (Schwarz et al., 2020) and so on.
  Texture Mapping. Texture mapping is an essential step in the computer graphics rendering pipeline. It defines a correspondence between a vertex on the 3D mesh and a pixel in the 2D texture image. To find such a mapping, FlexiStickers (Tzur & Tal, 2009) required users to specify a sparse set of correspondences. Bi (Bi et al., 2017) proposed a patch-based texture mapping method using the 3D shape and images from multiple views. Morreale (Morreale et al., 2021) used networks to represent 3D surfaces/shapes. Besides the above general texture mapping methods, some approaches focus on a specific object categories such as faces (Deng et al., 2018; Chen et al., 2019) and human bodies (Mir et al., 2020; Zhao et al., 2020). Recently, AtlasNet (Groueix et al., 2018) represented a 3D mesh as a collection of parametric surfaces; thus, texture mapping is trivial to obtain from a 2D parametric surface. A similar idea was adopted by Bednarik (Bednarik et al., 2020) where they introduced geometric constraints when learning the decomposition. More recently NeuTex (Xiang et al., 2021) aims to recover the texture of a subject using NeRF (Mildenhall et al., 2020). However, NeuTex uses a spherical UV domain without any geometric constraints. Therefore, it is not suitable for document unwarping. Moreover, since NeRF (Mildenhall et al., 2020) doesn’t learn an explicit geometry, NeuTex requires a coarse point-cloud to initialize the backward MLP. With an SDF based (Yariv et al., 2020) approach, our approach does not require such an initialization routine. We jointly learn the texture mapping and the geometry from scratch.
  Document Unwarping. Document unwarping is a special application of texture mapping: the 3D object is usually a rectangular piecewise-developable surface, and the texture is well-structured, containing straight text lines, (usually) rectangular text blocks and figures, etc. Previous work usually adopted a two-step methodology: 1) 3D surface estimation and 2) deformed surface flattening. The 3D surface of a deformed document can be estimated from shading (Wada et al., 1997), multi-view images (Ulges et al., 2004), text lines (Tian & Narasimhan, 2011), local character orientations (Meng et al., 2018), document boundaries (Koo et al., 2009), and learning-based strategies (Pumarola et al., 2018). Flattening the obtained 3D surface always involves an expensive optimization process under certain geometry constraints such as conformality (You et al., 2017) or isometries (Bartoli et al., 2015). Flattening could be easier if the obtained 3D shape had a low dimensional parameterization like Generalized Cylindrical Surface (GCS) (Kil et al., 2017). Some studies (Das et al., 2017; Liang et al., 2008; Meng et al., 2015) proposed to unwarp each patch on the surface individually and then stitch the unwarped patches together. In recent years, data-driven methods (Ma et al., 2018; Das et al., 2019; Li et al., 2019; Markovitz et al., 2020; Das et al., 2021) have addressed document unwarping by leveraging large-scale synthetic datasets. These datasets contain deformed document images and their corresponding ground-truth UV coordinates. Methods trained on synthetic images often suffer from generalization performance due to the domain gap between synthetic and real data. In this paper, we utilize neural rendering techniques to learn a surface parameterization of a deformed document. We simultaneously estimate both 3D shapes and UV coordinates with a cycle consistency loss and geometric constraints. By leveraging the information from multi-view images, the proposed method demonstrates better document unwarping performance compared to a previous state-of-the-art, Das et al. (2019).
sections.heading: 3 METHOD
sections.text: A schematic diagram of the proposed approach is shown in Fig. 2. We utilize a recent differentiable rendering method, IDR (Yariv et al., 2020) for surface reconstruction and jointly learn the texture mapping of the learned implicit surface using two MLPs. In Sec. 3.1 we first describe some preliminaries about surface parameterization and IDR.
sections.heading: 3.1 PRELIMINARIES
sections.text: Surface Parameterization. The problem of surface parameterization focuses on finding a bijective mapping F between a surface Z ∈ R3 and a polygonal domain Ω ∈ Rn. For a parametric or discrete surface representation, we can explicitly compute this mapping (Tzur & Tal, 2009) using constrained optimization. In contrast, implicit surfaces are represented as continuous functions and cannot be readily parameterized. In this paper, we propose to learn such bijective mapping between a learned
  implicit surface and a 2D planar domain Ω ∈ R2 using our proposed forward and backward MLPs. Ω is the texture space or UV space, parameterized using 2D UV coordinates t = (u, v). We can use any continuous parameterization function as the UV space. Since this work particularly focuses on document unwarping, we choose the UV space to be a regular 2D grid.
  Implicit Differentiable Rendering. Implicit Differentiable Rendering (Yariv et al., 2020) reconstructs the geometry of an object from multi-view images as the zero level set, Zθ of an MLP S,
  Zθ = {z ∈ R3 | S(z; θ) = 0} (1) where θ are the learnable parameters. To render the surface Zθ, IDR uses another MLP to model the radiance (RGB color) as a function of the surface point (zp), corresponding surface normal (np), view direction (vp) and a global geometry feature vector (gp):
  Cp = A(zp,np, vp, gp) (2) Here, Cp denote the predicted color at pixel p and A denotes the appearance MLP. The surface point is obtained by a sphere-tracing method (Hart, 1996) along the ray rp(τ) through pixel p. τ ∈ Rk denotes camera parameters of the scene. Additionally, IDR also presents a differentiable way to obtain a ray and geometry intersection point (ẑp) as a function of the camera ray. Although,the IDR can disentangle geometry and appearance, it only allows to re-texture a new geometry with a learned appearance MLP, A. Editing a texture or extracting a surface texture map is not possible in a vanilla IDR framework since no explicit texture mapping is learned.
sections.heading: 3.2 LEARNING SURFACE PARAMETERIZATION
sections.text: To learn a meaningful parameterization of the implicit surface Zθ, we represent the radiance at pixel p as a function of the UV space. To this end, we modify the IDR model (Eq. 2):
  Cp = Auv(tp, zp,np, vp, gp) (3) The texture parameterized appearance MLP is modeled as a function of the texture coordinate tp at surface point zp, corresponding to a pixel p. We can jointly train the surface MLP (S) and texture parameterized appearance MLP (Auv) using a pixel wise rendering loss between the predicted radiance (Cp) and ground-truth radiance (Cgtp ) at pixel p.
  Forward and backward texture parameterization. We represent the mapping between the 3D surface and 2D texture space using the forward function Fuv:
  z→ t. (4) The Fuv is modeled as an MLP. It is trained by mapping a ray-surface intersection point ẑp to its corresponding texture coordinate tp corresponding to a pixel p. Now to establish the bijective mapping (discussed in Sec. 3.1) between the surface and texture space we utilize a backward function Fz: t→ z. (5) Fz is an MLP that learns an inverse mapping between the texture and the 3D space. It is trained by mapping a texture coordinate tp to its corresponding ray-surface intersection point ẑp.
  Shape specific prior for Fuv . Jointly training the forward, backward and rendering network leads
  to the wrong UV mapping with local minima (see Fig. 3) where multiple ẑp map to a single texture coordinate. To avoid such degenerate cases, we initialize Fuv with a texture mapping prior, learned from a large dataset of UV mapped meshes. This learned prior (F̂uv) makes the learned texture mapping suitable for document unwarping. We assume the document shape to be a deformed quadrilateral and the corresponding UV space to be a regular grid (∈ [0.0, 1.0]). The top leftmost and the bottom rightmost 3D coordinate of the shape maps to (u, v) = (0, 0) and (u, v) = (1, 1) respectively. To learn F̂uv we utilize a collection of UV mapped document meshes from the Doc3D (Das et al., 2019) dataset and train an MLP with the same parameters as Fuv . For each scene, we use F̂uv to initialize the weights of Fuv and train jointly with S and Auv .
  Deformation constraints for Fz. Conformal map (Haker et al., 2000) allows a 3D domain to be mapped to a texture domain with low distortion satisfying the bijective property between domains. We use a conformality constraint for Fz to ensure the deformation properties mentioned above. We define the conformality constraint in terms of the metric tensor, J>J of the Fz, where J is the jacobian of Fz (Eq. 6):
  J = [ δFz δu δFz δv ] = [Du Dv] J>J = [ D>uDu D > uDv
  D>uDv D > v Dv
  ] = [ E F
  F G
  ] (6)
  The conformality constraint is defined as J>J = βI. Here β is a unknown local scaling function and I is the identity matrix. For developable surfaces which can be physically flattened without any stretch e.g. papers, β doesn’t vary across the parameterization space. Therefore, we consider a fixed global scale ([βgu, β g v ]) for the conformality constraint.
  Unwarping by sampling Fz. To unwarp an input image, we determine the pixel at p = (x, y) in the input image should be projected to (u, v) in the unwarped image. Here the unwarped image refers to the texture space. The coordinates (u, v) and p are associated by Fz and τ : For a (u, v) coordinate, its corresponding point in 3D is obtained by ẑ′p = Fz(u, v). Given the camera parameter τ , ẑ′p is projected to p in the input image. Thus for each pixel in the unwarped image, we can find its corresponding pixel in the input image which is all we need for unwarping.
sections.heading: 3.3 LOSS FUNCTIONS
sections.text: We use the rendering losses on the predicted color, Cp, and predicted document mask Mp at pixel p to train the geometry S. Here Mp ∈ {0, 1} refers to whether the pixel p is occupied (Mp = 1) by the shape or not (Mp = 0). We assume masks are provided as input. Additionally, we employ appropriate regularization losses to jointly train S, Auv , Fuv and Fz .
  Loss for S. Following IDR (Yariv et al., 2020), for each p we apply a sphere tracing (Hart, 1996) algorithm to find the intersection point of the ray rp(τ) and the surface Zθ. Given the ground-truth RGB color Cgtp and the predicted RGB color Cp, the RGB loss is defined as:
  Lrgb = 1 |P | ∑ p∈Pin ∥∥Cgtp − Cp∥∥1 (7) Where P is the set of pixels in the minibatch. The pixels Pin ⊂ P for which ray surface intersection has been found and Mp = 1. The mask loss is defined as:
  Lmask = 1 α|P | ∑
  p∈Pout
  CE(Mgtp ,Mp) (8)
  Here Pout = P \Pin, alpha is a tunable parameter and CE(.) is the cross-entropy loss. The value of Mp =Mp,α(θ, τ) is a differentiable function of the learned Zθ (Yariv et al., 2020). Additionally, to force Zθ to be a approximate signed distance function we use Eikonal Regularization (Gropp et al.,
  2020): Lek = Ez(‖∇zS(z; θ)‖ − 1)2 (9) where z denotes uniformly sampled points within a bounding box of the 3D domain.
  Loss for Fuv. Although we initialize Fuv with learned prior parameters, we constrain the predicted 2D texture coordinates during training in order to avoid non-uniform mapping of the 3D and the UV domain which can squeeze or stretch the warped texture (example in supplementary). We employ a Chamfer distance between the tp and uniformly sampled 2D points T ∈ [0, 1] to ensure Fuv approximately outputs U ∼ [0, 1]. This regularization term is defined as:
  Luv = CDp∈Pin(T , tp) (10) here CD(.) denotes the Chamfer distance and tp the predicted texture coordinates corresponding to ray-surface intersection points ẑp.
  Loss for Fz. ẑ′p is the output of Fz. Fz is trained with weighted regression loss between ẑp and ẑ′p:
  Lz = 1 |Pin| ∑ p∈Pin wp(ẑp − ẑ′p)2 (11)
  wp is a pre-calculated per-pixel weight based on the document mask (M ) which assigns higher value to the pixels at the boundary of the document. (More weight calculation details in Supplementary).
  Additionally, to constrain Fz to be a fixed scale conformal mapping (Bednarik et al., 2020). We employ three constraints on the elements of the metric tensor E, F and G defined in Eq. 6.
  LE = 1 |Pin| ∑ p∈Pin (Ep − Ẽ)2 LG = 1 |Pin| ∑ p∈Pin (Gp − G̃)2 LF = 1 |Pin| ∑ p∈Pin (Fp) 2
  Here Ẽ and G̃ is the mean of E and G.
  Our combined loss function is defined as: L = (Lrgb + γ1Lmask + γ2Lek)︸ ︷︷ ︸
  LS
  +ρLuv + (δ1Lz + δ2LE + δ3LG + δ4LF )︸ ︷︷ ︸ LT
  (12)
  Here γ, ρ and δ denote the hyperparameters associated with the losses.
sections.heading: 3.4 TRAINING DETAILS
sections.text: The surface MLP S(z, θ) consists of 8 layers with a hidden layer dimension of 128, with a skip connection to the middle layer (Park et al., 2019a). Following IDR (Yariv et al., 2020), S is initialized to produce an approximate SDF of a unit sphere. The rendering network Auv has 4 layers with hidden layer dimension of 512 and uses a sine activation function (Sitzmann et al., 2020) at each layer. Fuv and Fz share identical architecture with 8 layers with 512 dimensional hidden units and sine activation (Sitzmann et al., 2020). Following NeRF (Mildenhall et al., 2020), we use a k dimensional Fourier mapping (χk : R → R2k) to learn high frequency details in the shape, RGB and the UV space. For S, Auv we follow the setting of (Yariv et al., 2020), and set k = 6 and k = 4 respectively. For Fuv and Fz we empirically set number of Fourier bands k = 10. We start with an initial learning rate of 1e×−5 and train for 150K iterations by halving the learning rate after every 50K iterations. Initially, α is set to 50 and doubled during the training after every 50K iterations. We set γ1 = 100.0, γ2 = 0.1 and ρ = 0.001. δ1 is set to 0.001 for the initial 30K iterations. Afterward, δ1 is multiplied by a factor 2 at every 10K iterations for a maximum of 7 times. δ2, δ3 and δ4, are set to zero for the initial 100K iterations. Only Lz is sufficient to achieve a good texture to 3D mapping during the shape optimization phase. Afterwards we set δ2 = δ3 = 0.001 and δ4 = 0.01. The metric tensor calculation is implemented using auto-differentiation.
sections.heading: 4 EXPERIMENTAL RESULTS
sections.text: First, we quantitatively compare the proposed method with state-of-the-art document unwarping method DewarpNet (Das et al., 2019). Our quantitative and qualitative experiments are performed on 10 synthetic scenes and 10 real scenes. Second, we apply our method to texture editing. Last, we conduct ablation studies to demonstrate the effectiveness of our proposed loss functions.
sections.heading: 4.1 EVALUATION DATASET AND METRICS
sections.text: Our synthetic evaluation data consists of 10 scenes rendered using Blender following a rendering pipeline similar to Doc3D. Each scene consists of 50 random views sampled from a 45o solid angle in the upper hemisphere. The real-world evaluation data consists of 3 scenes from the dataset of (You et al., 2017), and 9 scenes captured by us. Each scene consists of 5-20 images per scene. We manually annotate the masks for each scene. To obtain camera poses for the real-world data, we utilize the COLMAP (Schönberger & Frahm, 2016) multi-view reconstruction pipeline. Both synthetic and real data include the document scan, as the unwarping ground-truth.
  We use image-based evaluation metrics for quantitative evaluation, including Local Distortion (LD) and Multi-Scale Structural Similarity (MS-SSIM). These are standard metrics used for document unwarping evaluation (Das et al., 2019; Ma et al., 2018). LD is based on dense SIFT flow (Liu et al., 2011) between the unwarped and scanned images. Image similarity metric, MS-SSIM (Wang et al., 2003) is based on local image statistics (mean and variance) of the unwarped and scanned (groundtruth) images calculated over multiple Gaussian pyramid scales. We use the same settings as (Das et al., 2019; Ma et al., 2018) for fair comparison.
sections.heading: 4.2 DOCUMENT UNWARPING
sections.text: The primary application of our learned forward and backward MLP is document unwarping. The quantitative comparison with the state-ofthe-art model (Das et al., 2019) is shown in Table 1 for the synthetic and real scenes. In terms of average performance of all the views (all views col. in Table 1) we improve the LD by ∼ 52% compared to (Das et al., 2019). Since we use multi-view images for training, our results are more consistent across all the views compared to DewarpNet, which is also a key reason for the significant improvement. We conjecture that (Das et al., 2019) as a single image unwarping method should perform well on simpler deformations and frontal view images. However, it is not always the case. In qualitative comparisons in Fig. 4, DewarpNet often gen-
  erates artifacts even for reasonably frontal views and simple deformations. Comparatively, our results are qualitatively superior.
  We also report in a stricter evaluation scenario (best view column of Table 1) where we compare our results with the best possible numerical results achieved by DewarpNet from a single view in a scene. We perform better than DewarpNet in 91.2% of all views, however when the best view can be selected our method do slightly worse in 7 scenes. This ‘stricter’ setting shows quantitatively competitive results compared to DewarpNet with a oracle (practically challenging) view selector.The choice of the best unwarped result is often subjective. For a more comprehensive comparison, we qualitatively compare the best results of DewarpNet with our results across 6 scenes in Fig. 5. These 6 scenes are chosen among the 7 scenes for which DewarpNet achieves a better quantitative result than the proposed approach for at least one view. In Fig. 5 our results are clearly better than the DewarpNet in all cases, with straighter lines and better rectified structure. The evaluation scores do not accurately reflect the improvement due to the sensitivity of LD and MSSIM to subtle perceptually unimportant global transformations, such as translation of the image by few pixels. However, such transformations do not affect the visual quality or readability of the unwarped results. More discussion and qualitative comparison is available in supplementary material.
  DewarpNet Proposed
  ED ↓ CER (std) ↓ WER (std) ↓ ED ↓ CER (std) ↓ WER (std) ↓ Mean 798.30 0.2827 (0.12) 0.4646 (0.17) 600.78 0.2122 (0.10) 0.3568 (0.11)
  Table 2: Comparison of OCR error metrics: We improve the OCR performance of Das et al. (2019) by∼25% in terms of Edit Distance (ED), Character Error Rate (CER), and Word Error Rate (WER).
  The quantitative comparison for real scenes are reported in Table 1 (bottom). We achieve better results in terms of mean and best evaluation score than DewarpNet in 2 out of 3 scenes. We notice that the evaluation results are a little worse for the real scenes than synthetic scenes due to the fewer available views (5-10 compared to 50). Moreover, there are cases like Real 6, which do not have sufficient texture. Such data are a failure case of IDR since there is insufficient information to reconstruct the 3D shape. As a result of the poor 3D shape, our texture parameterization network produces an inferior unwarping result (More details are available in Supplementary). We also report qualitative comparisons with You et al. (2017) and Das et al. (2019) on additional real documents in supplementary.
  OCR Evaluation. We also evaluated the OCR performance on 5 real scenes across 77 images in Table 2. We use Edit Distance (ED) (Miller et al., 2009), Character Error Rate (CER) and Word Error Rate (WER) as our evaluation metrics. ED is defined as the total number of substitutions (s), insertions (i) and deletions (d) required to obtain the reference text, given the recognized text. The reference text is obtained by running the OCR algorithm on the scanned ground-truth image of each document. CER is defined as: (s+ i+ d)/N where N is the number of characters in the reference text. We use Tesseract 4.1.1 based LSTM OCR engine for this experiment. Our unwarped results reduce the ED, CER and WER by ∼25%. This improvement proves our unwarped results are more suitable for downstream applications tasks like OCR.
  Texture Editing. In addition to document unwarping, our proposed forward and backward MLP can also be used for high quality texture editing. We show two texture editing examples in Fig. 1. We use the backward MLP to unwarp the texture from the input image, then we edit the texture and warp it back to image space using the learned forward MLP. (More details in Supplementary).
  Ablation Study. We ablate how loss terms Lz , LE , LF , and LG affect the unwarping results. We train Fuv and Fz with different combinations of these loss terms and report the mean MSSIM and LD in Table 3 (appendix). Qualitative results for one scene are shown in Fig. 6 (appendix).
sections.heading: 5 TRAINING TIME, GENERALIZABILITY AND FUTURE WORK
sections.text: Our proposed method for a scene can be trained in approximately 18 hours for 448× 448 resolution images using a single Titan Xp GPU. The current training time per scene is very high compared to DewarpNet’s inference time which makes it unsuitable for real time applications. However, this is a fast growing field and there are multiple other works that are focusing on improving the speed and generalization abilities (Garbin et al., 2021; Bergman et al., 2021) of neural rendering. Therefore, obtaining a faster training scheme is considered as a future work.
  Our method can be applied to fabrics, which are very similar to papers and lead to practical applications of texture editing. However, none of the current 3D garment/fabric datasets (Patel et al., 2020) can be easily adapted to train the F̂uv prior. For more complex UV spaces (e.g., texture atlas), learning the prior may require decomposing the shape to multiple simple UV maps. The proper way to do this is beyond the scope of this paper, however we believe it’s an exciting future work. As importantly, in this paper, we have introduced a number of domain specific strong constraints that suit the rectangular paper shape. These constraints improve empirical results. More general objects will require different constraints e.g., spherical UV domain, local scaling of the conformal map etc.
sections.heading: 6 CONCLUSIONS
sections.text: We have introduced an end-to-end trainable architecture that can simultaneously learn texture parameterized 3D shapes from multi-view images. This is the first work to learn surface parameterization of an implicit neural representation to the best of our knowledge. We have demonstrated the applicability of our approach on multiple synthetic and real scenes for the task of document unwarping and document texture editing. We want to extend this method to learn surface parameterization for more complex shapes such as faces or general 3D objects in future work.
sections.heading: 7 ETHICS STATEMENT
sections.text: Texture editing application of our proposed approach can have both positive and negative societal impact. On the positive side, real document images can be gracefully redacted to protect sensitive information. On the contrary, it can be potentially used for editing real documents and change the content to commit fraud and spread misinformation.
sections.heading: 8 REPRODUCIBILITY STATEMENT
sections.text: We believe our results are reproducible by following the training details in Section 3.4 of main submission and Section 2, 3, 4, 5 from the supplementary material.
sections.heading: A APPENDIX
sections.text: A.1 ABLATION STUDY
  We ablate how loss terms Lz , LE , LF , and LG affect the unwarping results. We train Fuv and Fz with different combinations of these loss terms and report the mean MSSIM and LD in Table 3. Qualitative results for one scene are shown in Fig. 6 (appendix). In the basic version (listed as b), no conformality constraints (LE , LF and LG) are used and wp = 1 in Lz . The b+w version introduces a weighting function that assigns a higher value to wp if a pixel is closer to the document boundary. Introducing this loss improves the boundary; notice the white margin at the top and bottom in the second column of Fig. 6. Introducing the conformality constraints alleviates the unusual stretch in the texture and improves smoothness (Fig. 6, col. 3). Using both improves the boundary and the texture smoothness (Fig. 6, col. 4) and achieves the best result.
  (a) (b) (c)
  (d) (e) (f)