summaries: This work introduces a differentiable points (+normals) to mesh layer based on a differentiable formulation of Poisson surface reconstruction. First, the proposed method is validated in point cloud optimisation tasks where the objective function is expressed w.r.t surface meshes. Then, the method is proposed as a way to represent 3D shapes with oriented point cloud in a learning setting. Crucially, this results in a shape parameterization, dubbed Shape As Points (SAP), that is more lightweight and faster than neural implicit fields, while still yielding watertight shapes and being able to handle arbitrary topology.

summaries: This paper proposes a differentiable adaptation of the Poisson Surface Reconstruction algorithm for oriented point sets. It discretizes the Poisson equation over a 3D grid and use spectral methods to find the solution for indicator function values over this 3D grid. The final grid is fed to marching cubes algorithm to obtain the final mesh. The differentiability of the approach allows to use the proposed method in various optimization-based applications from which the authors consider optimization-based and learning-based 3D reconstruction tasks. Proposed models are evaluated on datasets with different level of details and noise present in the inputs and compared to both classical and more recent learning-based methods.

summaries: This work proposes a novel shape representation using point clouds. This is enabled by unique differentiable poisson solver layer that enables to convert the oriented point clouds to full volume representing the shape. I think the main contribution of this work is the spectral technique that solves poisson surface reconstruction problem, which can be efficiently integrated into neural workflows. While this work mostly focuses on surface reconstruction, I believe this representation can become broadly applicable to various problem domains.

summaries: The paper formulates a differentiable Poisson Surface Reconstruction layer, which maps an oriented point cloud to an indicator function over the interior of the object. The Poisson solver is done in the spectral domain on the GPU for computational efficiency. It shares many of its advantages with other neural implicit representations, but because it produces an indicator function over the whole space, there is no need to query a network over an entire voxel grid.   The layer is incorporated into two tasks: an optimization-based 3D shape reconstruction that fits a spherical point cloud initialization to a target point cloud, and a learning-based 3D reconstruction task that matches noisy point cloud input to a ground truth mesh. The non-differentiable step of Marching Cubes in these pipelines has its gradient approximated with the inverse surface normal, which is quite natural. The experimental results seem good, and beat out the existing state-of-the-art, and an ablation study is performed as well on the learning-based task to investigate the form of the feature extraction network there.

abstractText: In recent years, neural implicit representations gained popularity in 3D reconstruction due to their expressiveness and flexibility. However, the implicit nature of neural implicit representations results in slow inference time and requires careful initialization. In this paper, we revisit the classic yet ubiquitous point cloud representation and introduce a differentiable point-to-mesh layer using a differentiable formulation of Poisson Surface Reconstruction (PSR) that allows for a GPUaccelerated fast solution of the indicator function given an oriented point cloud. The differentiable PSR layer allows us to efficiently and differentiably bridge the explicit 3D point representation with the 3D mesh via the implicit indicator field, enabling end-to-end optimization of surface reconstruction metrics such as Chamfer distance. This duality between points and meshes hence allows us to represent shapes as oriented point clouds, which are explicit, lightweight and expressive. Compared to neural implicit representations, our Shape-As-Points (SAP) model is more interpretable, lightweight, and accelerates inference time by one order of magnitude. Compared to other explicit representations such as points, patches, and meshes, SAP produces topology-agnostic, watertight manifold surfaces. We demonstrate the effectiveness of SAP on the task of surface reconstruction from unoriented point clouds and learning-based reconstruction.

authors:
M. Atzmon
Y. Lipman

title: Sal: Sign agnostic learning of shapes from raw data

venue: In Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR),

authors:
A. Badki
O. Gallo
J. Kautz
P. Sen

title: Meshlet priors for 3d mesh reconstruction

authors:
Y. Ben-Shabat
S. Gould

title: Deepfit: 3d surface fitting via neural network weighted least squares

venue: In Proc. of the European Conf. on Computer Vision (ECCV),

authors:
Y. Ben-Shabat
M. Lindenbaum
A. Fischer

title: Nesti-net: Normal estimation for unstructured 3d point clouds using convolutional neural networks

authors:
F. Bernardini
J. Mittleman
H. Rushmeier
C. Silva
G. Taubin

title: The ball-pivoting algorithm for surface reconstruction

venue: IEEE Trans. on Visualization and Computer Graphics (VCG),

authors:
F. Bogo
J. Romero
G. Pons-Moll
M.J. Black

title: Dynamic FAUST: registering human bodies in motion

authors:
C. Canuto
M.Y. Hussaini
A. Quarteroni
T.A. Zang

title: Spectral methods: fundamentals in single domains

venue: Springer Science & Business Media,

authors:
R. Chabra
J.E. Lenssen
E. Ilg
T. Schmidt
J. Straub
S. Lovegrove
R. Newcombe

title: Deep local shapes: Learning local sdf priors for detailed 3d reconstruction

authors:
A.X. Chang
T.A. Funkhouser
L.J. Guibas
P. Hanrahan
Q. Huang
Z. Li
S. Savarese
M. Savva
S. Song
H. Su
J. Xiao
L. Yi
F. Yu

title: Shapenet: An information-rich 3d model repository

authors:
Z. Chen
H. Zhang

title: Learning implicit fields for generative shape modeling

authors:
C.B. Choy
D. Xu
J. Gwak
K. Chen
S. Savarese

title: 3d-r2n2: A unified approach for single and multi-view 3d object reconstruction

authors:
A. Dai
C. Diller
M. Nießner

title: SG-NN: sparse generative neural networks for self-supervised scene completion of RGB-D scans

title: Sg-nn: Sparse generative neural networks for self-supervised scene completion of rgb-d scans

authors:
A. Dai
D. Ritchie
M. Bokeloh
S. Reed
J. Sturm
M. Nießner

title: Scancomplete: Large-scale scene completion and semantic segmentation for 3d scans

authors:
H. Edelsbrunner
E.P. Mücke

title: Three-dimensional alpha shapes

venue: ACM Trans. on Graphics,

authors:
P. Erler
P. Guerrero
S. Ohrhallinger
M. Wimmer
N.J. Mitra

title: Points2surf: Learning implicit surfaces from point cloud patches

authors:
H. Fan
H. Su
L.J. Guibas

title: A point set generation network for 3d object reconstruction from a single image

authors:
K. Genova
F. Cole
A. Sud
A. Sarna
T.A. Funkhouser

title: Local deep implicit functions for 3d shape

authors:
A. Gropp
L. Yariv
N. Haim
M. Atzmon
Y. Lipman

title: Implicit geometric regularization for learning shapes

venue: In Proc. of the International Conf. on Machine learning (ICML),

authors:
T. Groueix
M. Fisher
V.G. Kim
B.C. Russell
M. Aubry

title: AtlasNet: A papier-mâché approach to learning 3d surface generation

authors:
P. Guerrero
Y. Kleiman
M. Ovsjanikov
N.J. Mitra

title: Pcpnet learning local shape properties from raw point clouds

venue: In Computer Graphics Forum,

authors:
K. Gupta
M. Chandraker

title: Neural mesh flow: 3d manifold mesh generation via diffeomorphic flows

venue: In Advances in Neural Information Processing Systems (NeurIPS),

authors:
R. Hanocka
G. Metzer
R. Giryes
D. Cohen-Or

title: Point2mesh: a self-prior for deformable meshes

venue: In ACM Trans. on Graphics,

authors:
C. Jiang
J. Huang
A. Tagliasacchi
L.J. Guibas

title: Shapeflow: Learnable deformation flows among 3d shapes

authors:
C. Jiang
P. Marcus

title: Hierarchical detail enhancing mesh-based shape generation with 3d generative adversarial network

venue: arXiv preprint arXiv:1709.07581,

authors:
C. Jiang
A. Sud
A. Makadia
J. Huang
M. Nießner
T. Funkhouser

title: Local implicit grid representations for 3d scenes

authors:
C.M. Jiang
J. Huang
K. Kashinath
Prabhat
P. Marcus
M. Nießner

title: Spherical cnns on unstructured grids

venue: In Proc. of the International Conf. on Learning Representations (ICLR),

authors:
M.M. Kazhdan
M. Bolitho
H. Hoppe

title: Poisson surface reconstruction

venue: In Proceedings of the Fourth Eurographics Symposium on Geometry Processing,

authors:
M.M. Kazhdan
H. Hoppe

title: Screened poisson surface reconstruction

authors:
D.P. Kingma
J. Ba

title: Adam: A method for stochastic optimization

authors:
J.E. Lenssen
C. Osendorfer
J. Masci

title: Deep iterative surface normal estimation

authors:
J. Li
A.O. Hero

title: A spectral method for solving elliptic equations for surface reconstruction and 3d active contours

venue: In Proc. IEEE International Conf. on Image Processing (ICIP),

authors:
Y. Liao
S. Donne
A. Geiger

title: Deep marching cubes: Learning explicit surface representations

authors:
S. Lionar
D. Emtsev
D. Svilarkovic
S. Peng

title: Dynamic plane convolutional occupancy networks

venue: In Proc. of the IEEE Winter Conference on Applications of Computer Vision (WACV),

authors:
S. Liu
Y. Zhang
S. Peng
B. Shi
M. Pollefeys
Z. Cui

title: Dist: Rendering deep implicit signed distance function with differentiable sphere tracing

authors:
W.E. Lorensen
H.E. Cline

title: Marching cubes: A high resolution 3d surface construction algorithm

authors:
Q. Ma
S. Saito
J. Yang
S. Tang
M.J. Black

title: Scale: Modeling clothed humans with a surface codec of articulated local elements

authors:
Q. Ma
J. Yang
S. Tang
M.J. Black

title: The power of points for modeling humans in clothing

venue: In Proc. of the IEEE International Conf. on Computer Vision (ICCV),

authors:
R. Martin-Brualla
N. Radwan
M.S.M. Sajjadi
J.T. Barron
A. Dosovitskiy
D. Duckworth

title: NeRF in the Wild: Neural Radiance Fields for Unconstrained Photo Collections

authors:
L. Mescheder
M. Oechsle
M. Niemeyer
S. Nowozin
A. Geiger

title: Occupancy networks: Learning 3d reconstruction in function space

authors:
M. Meshry
D.B. Goldman
S. Khamis
H. Hoppe
R. Pandey
N. Snavely
R. Martin- Brualla

title: Neural rerendering in the wild

authors:
G. Metzer
R. Hanocka
D. Zorin
R. Giryes
D. Panozzo
D. Cohen-Or

title: Orienting point clouds with dipole propagation

authors:
M. Mihajlovic
S. Weder
M. Pollefeys
M.R. Oswald

title: Deepsurfels: Learning online appearance fusion

authors:
M. Mihajlovic
Y. Zhang
M.J. Black
S. Tang

title: LEAP: Learning articulated occupancy of people

authors:
B. Mildenhall
P.P. Srinivasan
M. Tancik
J.T. Barron
R. Ramamoorthi
R. Ng

title: NeRF: Representing scenes as neural radiance fields for view synthesis

authors:
M. Niemeyer
A. Geiger

title: Giraffe: Representing scenes as compositional generative neural feature fields

authors:
M. Niemeyer
L. Mescheder
M. Oechsle
A. Geiger

title: Differentiable volumetric rendering: Learning implicit 3d representations without 3d supervision

authors:
M. Oechsle
L. Mescheder
M. Niemeyer
T. Strauss
A. Geiger

title: Texture fields: Learning texture representations in function space

authors:
M. Oechsle
S. Peng
A. Geiger

title: Unisurf: Unifying neural implicit surfaces and radiance fields for multi-view reconstruction

authors:
J.J. Park
P. Florence
J. Straub
R.A. Newcombe
S. Lovegrove

title: Deepsdf: Learning continuous signed distance functions for shape representation

authors:
A. Paszke
S. Gross
F. Massa
A. Lerer
J. Bradbury
G. Chanan
T. Killeen
Z. Lin
N. Gimelshein
L. Antiga
A. Desmaison
A. Kopf
E. Yang
Z. DeVito
M. Raison
A. Tejani
S. Chilamkurthy
B. Steiner
L. Fang
J. Bai
S. Chintala

title: Pytorch: An imperative style, high-performance deep learning library

authors:
S. Peng
M. Niemeyer
L. Mescheder
M. Pollefeys
A. Geiger

title: Convolutional occupancy networks

authors:
C.R. Qi
H. Su
K. Mo
L.J. Guibas

title: Pointnet: Deep learning on point sets for 3d classification and segmentation

authors:
C.R. Qi
L. Yi
H. Su
L.J. Guibas

title: Pointnet++: Deep hierarchical feature learning on point sets in a metric space

authors:
C. Reiser
S. Peng
Y. Liao
A. Geiger

title: Kilonerf: Speeding up neural radiance fields with thousands of tiny mlps

authors:
E. Remelli
A. Lukoianov
S.R. Richter
B. Guillard
T. Bagautdinov
P. Baque
P. Fua

title: Meshsdf: Differentiable iso-surface extraction

authors:
V. Sitzmann
J.N. Martel
A.W. Bergman
D.B. Lindell
G. Wetzstein

title: Implicit neural representations with periodic activation functions

authors:
V. Sitzmann
J. Thies
F. Heide
M. Nießner
G. Wetzstein
M. Zollhöfer

title: Deepvoxels: Learning persistent 3d feature embeddings

authors:
M. Tancik
P. Srinivasan
B. Mildenhall
S. Fridovich-Keil
N. Raghavan
U. Singhal
R. Ramamoorthi
J. Barron
R. Ng

title: Fourier features let networks learn high frequency functions in low dimensional domains

authors:
S. Van der Walt
J.L. Schönberger
J. Nunez-Iglesias
F. Boulogne
J.D. Warner
N. Yager
E. Gouillart
T. Yu

title: scikit-image: image processing in python

venue: PeerJ,

authors:
N. Wang
Y. Zhang
Z. Li
Y. Fu
W. Liu
Y.-G. Jiang

title: Pixel2mesh: Generating 3d mesh models from single rgb images

authors:
S. Wang
M. Mihajlovic
Q. Ma
A. Geiger
S. Tang

title: Metaavatar: Learning animatable clothed human models from few depth images

authors:
W. Wang
Q. Xu
D. Ceylan
R. Mech
U. Neumann

title: Disn: Deep implicit surface network for high-quality single-view 3d reconstruction

authors:
Y. Wang
Y. Sun
Z. Liu
S.E. Sarma
M.M. Bronstein
J.M. Solomon

title: Dynamic graph cnn for learning on point clouds

authors:
F. Williams
T. Schneider
C. Silva
D. Zorin
J. Bruna
D. Panozzo

title: Deep geometric prior for surface reconstruction

authors:
J. Wu
C. Zhang
T. Xue
B. Freeman
J. Tenenbaum

title: Learning a probabilistic latent space of object shapes via 3d generative-adversarial modeling

authors:
G. Yang
X. Huang
Z. Hao
M. Liu
S.J. Belongie
B. Hariharan

title: Pointflow: 3d point cloud generation with continuous normalizing flows

authors:
Y. Yang
C. Feng
Y. Shen
D. Tian

title: Foldingnet: Point cloud auto-encoder via deep grid deformation

authors:
Z. Yang
Y. Chai
D. Anguelov
Y. Zhou
P. Sun
D. Erhan
S. Rafferty
H. Kretzschmar

title: Surfelgan: Synthesizing realistic sensor data for autonomous driving

authors:
L. Yariv
Y. Kasten
D. Moran
M. Galun
M. Atzmon
B. Ronen
Y. Lipman

title: Multiview neural surface reconstruction by disentangling geometry and appearance

authors:
Q. Zhou
A. Jacobson

title: Thingi10k: A dataset of 10,000 3d-printing models

venue: arXiv preprint arXiv:1605.04797,

authors:
Q.-Y. Zhou
J. Park
V. Koltun

title: Open3D: A modern library for 3D data processing

heading: 1 Introduction

text:
Shape representations are central to many of the recent advancements in 3D computer vision and computer graphics, ranging from neural rendering [41,45,48,55,58] to shape reconstruction [10,26,40, 47, 50, 52, 70]. While conventional representations such as point clouds and meshes are efficient and well-studied, they also suffer from several limitations: Point clouds are lightweight and easy to obtain, but do not directly encode surface information. Meshes, on the other hand, are usually restricted to fixed topologies. More recently, neural implicit representations [10, 40, 50] have shown promising results for representing geometry due to their flexibility in encoding varied topologies, and their easy integration with differentiable frameworks. However, as such representations implicitly encode surface information, extracting the underlying surface is typically slow as they require numerous network evaluations in 3D space for extracting complete surfaces using marching cubes [10, 40, 50], or along rays for intersection detection in the context of volumetric rendering [45, 47, 49, 70].
In this work, we introduce a novel Poisson solver which performs fast GPU-accelerated Differentiable Poisson Surface Reconstruction (DPSR) and solves for an indicator function from an oriented point cloud in a few milliseconds. Thanks to the differentiablility of our Poisson solver, gradients from a loss on the output mesh or a loss on the intermediate indicator grid can be efficiently backpropagated to update the oriented point cloud representation. This differential bridge between points, indicator ∗Work done while at UC Berkeley. †Corresponding authors.
35th Conference on Neural Information Processing Systems (NeurIPS 2021).
functions, and meshes allows us to represent shapes as oriented point clouds. We therefore call this shape representation Shape-As-Points (SAP). Compared to existing shape representations, Shape-AsPoints has the following advantages (see also Table 1):
Efficiency: SAP has a low memory footprint as it only requires storing a collection of oriented point samples at the surface, rather than volumetric quantities (voxels) or a large number of network parameters for neural implicit representations. Using spectral methods, the indicator field can be computed efficiently (12 ms at 1283 resolution3), compared to the typical rather slow query time of neural implicit networks (330 ms using [40] at the same resolution). Accuracy: The resulting mesh can be generated at high resolutions, is guaranteed to be watertight, free from self-intersections and also topology-agnostic. Initialization: It is easy to initialize SAP with a given geometry such as template shapes or noisy observations. In contrast, neural implicit representations are harder to initialize, except for few simple primitives like spheres [1]. See supplementary for more discussions.
To investigate the aforementioned properties, we perform a set of controlled experiments. Moreover, we demonstrate state-of-the-art performance in reconstructing surface geometry from unoriented point clouds in two settings: an optimization-based setting that does not require training and is applicable to a wide range of shapes, and a learning-based setting for conditional shape reconstruction that is robust to noisy point clouds and outliers. In summary, the main contributions of this work are:
• We present Shape-As-Points, a novel shape representation that is interpretable, lightweight, and yields high-quality watertight meshes at low inference times.
• The core of the Shape-As-Points representation is a versatile, differentiable and generalizable Poisson solver that can be used for a range of applications.
• We study various properties inherent to the Shape-As-Points representation, including inference time, sensitivity to initialization and topology-agnostic representation capacity.
• We demonstrate state-of-the-art reconstruction results from noisy unoriented point clouds at a significantly reduced computational budget compared to existing methods.
Code is available at https://github.com/autonomousvision/shape_as_points.

heading: 2 Related Work

heading: 2.1 3D Shape Representations

text:
3D shape representations are central to 3D computer vision and graphics. Shape representations can be generally categorized as being either explicit or implicit. Explicit shape representations and learning algorithms depending on such representations directly parameterize the surface of the geometry, either as a point cloud [17, 38, 53, 54, 64, 67], parameterized mesh [22, 24, 27, 61] or surface patches [2, 20, 37, 43, 65, 68, 69]. Explicit representations are usually lightweight and require few parameters to represent the geometry, but suffer from discretization, the difficulty to represent
3On average, our method requires 12 ms for computing a 1283 indicator grid from 15K points on a single NVIDIA GTX 1080Ti GPU. Computing a 2563 indicator grid requires 140 ms.
watertight surfaces (point clouds, surface patches), or are restricted to a pre-defined topology (mesh). Implicit representations, in contrast, represent the shape as a level set of a continuous function over a discretized voxel grid [14, 25, 33, 66] or more recently parameterized as a neural network, typically referred to as neural implicit functions [10, 40, 50]. Neural implicit representations have been successfully used to represent geometries of objects [10, 16, 18, 35, 40, 44, 47, 50, 57, 59, 62, 63] and scenes [8, 26, 34, 46, 52, 57]. Additionally, neural implicit functions are able to represent radiance fields which allow for high-fidelity appearance and novel view synthesis [39,45]. However, extracting surface geometry from implicit representations typically requires dense evaluation of multi-layer perceptrons, either on a volumetric grid or along rays, resulting in slow inference time. In contrast, SAP efficiently solves the Poisson Equation during inference by representing the shape as an oriented point cloud.

heading: 2.2 Optimization-based 3D Reconstruction from Point Clouds

text: Several works have addressed the problem of inferring continuous surfaces from a point cloud. They tackle this task by utilizing basis functions, set properties of the points, or neural networks. Early works in shape reconstruction from point clouds utilize the convex hull or alpha shapes for reconstruction [15]. The ball pivoting algorithm [5] leverages the continuity property of spherical balls of a given radius. One of the most popular techniques, Poisson Surface Reconstruction (PSR) [28,29], solves the Poisson Equation and inherits smoothness properties from the basis functions used in the Poisson Equation. However, PSR is sensitive to the normals of the input points which must be inferred using a separate preprocessing step. In contrast, our method does not require any normal estimation and is thus more robust to noise. More recent works take advantage of the continuous nature of neural networks as function approximators to fit surfaces to point sets [19, 23, 42, 65]. However, these methods tend to be memory and computationally intensive, while our method yields high-quality watertight meshes in a few milliseconds.

heading: 2.3 Learning-based 3D Reconstruction from Point Clouds

text: Learning-based approaches exploit a training set of 3D shapes to infer the parameters of a reconstruction model. Some approaches focus on local data priors [2, 26] which typically result in better generalization, but suffer when large surfaces must be completed. Other approaches learn objectlevel [33, 40, 50] or scene-level priors [12, 13, 26, 52]. Most reconstruction approaches directly reconstruct a meshed surface geometry, though some works [3, 4, 21, 31] first predict point set normals to subsequently reconstruct the geometry via PSR [28, 29]. However, such methods fail to handle large levels of noise, since they are unable to move points or selectively ignore outliers. In contrast, our end-to-end approach is able to address this issue by either moving outlier points to the actual surface or by selectively muting outliers either by forming paired point clusters that self-cancel or reducing the magnitude of the predicted normals which controls their influence on the reconstruction.

heading: 3 Method

text: At the core of the Shape-As-Points representation is a differentiable Poisson solver, which can be used for both optimization-based and learning-based surface estimation. We first introduce the Poisson solver in Section 3.1. Next, we investigate two applications using our solver: optimization-based 3D reconstruction (Section 3.2) and learning-based 3D reconstruction (Section 3.3).

heading: 3.1 Differentiable Poisson Solver

text:
The key step in Poisson Surface Reconstruction [28, 29] involves solving the Poisson Equation. Let x ∈ R3 denote a spatial coordinate and n ∈ R3 denote its corresponding normal. The Poisson Equation arises from the insight that a set consisting of point coordinates and normals {p = (c,n)} can be viewed as samples of the gradient of the underlying implicit indicator function χ(x) that describes the solid geometry. We define the normal vector field as a superposition of pulse functions v(x) = ∑ (ci,ni)∈{p} δ(x − ci,ni), where δ(x,n) = {n if x = 0 and 0 otherwise}. By applying the divergence operator, the variational problem transforms into the standard Poisson equation:
∇2χ := ∇ · ∇χ = ∇ · v (1)
In order to solve this set of linear Partial Differential Equations (PDEs), we discretize the function values and differential operators. Without loss of generality, we assume that the normal vector field v and the indicator function χ are sampled at r uniformly spaced locations along each dimension. Denote the spatial dimensionality of the problem to be d. Without loss of generality, we consider
the three dimensional case where n := r × r × r for d = 3. We have the indicator function χ ∈ Rn, the point normal field v ∈ Rn×d, the gradient operator ∇ : Rn 7→ Rn×d, the divergence operator (∇·) : Rn×d 7→ Rn, and the derived laplacian operator ∇2 := ∇ · ∇ : Rn 7→ Rn. Under such a discretization scheme, solving for the indicator function amounts to solving the linear system by inverting the divergence operator subject to boundary conditions of surface points having zero level set. Following [28], we fix the overall scale to m = 0.5 at x = 0:
χ = (∇2)−1∇ · v s.t. χ|x∈{c} = 0 and abs(χ|x=0) = m (2)
Point Rasterization: We obtain the uniformly discretized point normal field v by rasterizing the point normals onto a uniformly sampled voxel grid. We can differentiably perform point rasterization via inverse trilinear interpolation, similar to the approach in [28, 29]. We scatter the point normal values to the voxel grid vertices, weighted by the trilinear interpolation weights. The point rasterization process has O(n) space complexity, linear with respect to the number of grid cells, and O(N) time complexity, linear with respect to the number of points. See supplementary for details.
Spectral Methods for Solving PSR: In contrast to the finite-element approach taken in [28,29], we solve the PDEs using spectral methods [7]. While spectral methods are commonly used in scientific computing for solving PDEs and in some cases applied to computer vision problems [32], we are the first to apply them in the context of Poisson Surface Reconstruction. Unlike finite-element approaches that depend on irregular data structures such as octrees or tetrahedral meshes for discritizing space, spectral methods can be efficently solved over a uniform grid as they leverage highly optimized Fast Fourier Transform (FFT) operations that are well supported for GPUs, TPUs, and mainstream deep learning frameworks. Spectral methods decompose the original signal into a linear sum of functions represented using the sine / cosine basis functions whose derivatives can be computed analytically. This allows us to easily approximate differential operators in spectral space. We denote the spectral domain signals with a tilde symbol, i.e., ṽ = FFT(v). We first solve for the unnormalized indicator function χ′, not accounting for boundary conditions
χ′ = IFFT(χ̃) χ̃ = g̃σ,r(u) iu · ṽ −2π‖u‖2
g̃σ,r(u) = exp ( − 2σ 2‖u‖2
r2
) (3)
where the spectral frequencies are denoted as u := (u, v, w) ∈ Rn×d corresponding to the x, y, z spatial dimensions, and IFFT(χ̃) represents the inverse fast Fourier transform of χ̃. g̃σ,r(u) is a Gaussian smoothing kernel of bandwidth σ at grid resolution r in the spectral domain. The Gaussian kernel is used to mitigate ringing effects as a result of the Gibbs phenomenon from rasterizing the point normals. We denote the element-wise product as : Rn × Rn 7→ Rn, the L2-norm as ‖ · ‖2 : Rn×d 7→ Rn, and the dot product as (·) : Rn×d × Rn×d 7→ Rn. Finally, we subtract by the mean of the indicator function at the point set and scale the indicator function to obtain the solution to the PSR problem in Eqn. 2:
χ = m
abs(χ′|x=0)︸ ︷︷ ︸ scale
( χ′ − 1 |{c}| ∑
c∈{c}
χ′|x=c )
︸ ︷︷ ︸ subtract by mean
(4)
A detailed derivation of our differentiable PSR solver is provided in the supplementary material.

heading: 3.2 SAP for Optimization-based 3D Reconstruction

text:
We can use the proposed differentiable Poisson solver for various applications. First, we consider the classical task of surface reconstruction from unoriented point clouds. The overall pipeline for this setting is illustrated in Fig. 1 (top). We now provide details about each component.
Forward pass: It is natural to initialize the oriented 3D point cloud serving as 3D shape representation using the noisy 3D input points and corresponding (estimated) normals. However, to demonstrate the flexibility and robustness of our model, we purposefully initialize our model using a generic 3D sphere with radius r in our experiments. Given the orientated point cloud, we apply our Poisson solver to obtain an indicator function grid, which can be converted to a mesh using Marching Cubes [36].
Backward pass: For every point pmesh sampled from the meshM, we calculate a bi-directional L2 Chamfer Distance LCD with respect to the input point cloud. To backpropagate the loss LCD through
pmesh to point p in our source oriented point cloud, we decompose the gradient using the chain rule:
∂LCD ∂p = ∂LCD ∂pmesh ∂pmesh ∂χ ∂χ ∂p (5)
All terms in (5) are differentialable except for the middle one ∂pmesh∂χ which involves Marching Cubes. However, this gradient can be effectively approximated by the inverse surface normal [56]:
∂pmesh ∂χ = −nmesh (6)
where nmesh is the normal of the point pmesh. Different from MeshSDF [56] that uses the gradients to update the latent code of a pretrained implicit shape representation, our method updates the source point cloud using the proposed differentiable Poisson solver.
Resampling: To increase the robustness of the optimization process, we uniformly resample points and normals from the largest mesh component every 200 iterations, and replace all points in the original point clouds with the resampled ones. This resampling strategy eliminates outlier points that drift away during the optimization, and enforces a more uniform distribution of points. We provide an ablation study in supplementary.
Coarse-to-fine: To further decrease run-time, we consider a coarse-to-fine strategy during optimization. More specifically, we start optimizing at an indicator grid resolution of 323 for 1000 iterations, from which we obtain a coarse shape. Next, we sample from this coarse mesh and continue optimization at a resolution of 643 for 1000 iterations. We repeat this process until we reach the target resolution (2563) at which we acquire the final output mesh. See also supplementary.

heading: 3.3 SAP for Learning-based 3D Reconstruction

text:
We now consider the learning-based 3D reconstruction setting in which we train a conditional model that takes a noisy, unoriented point cloud as input and outputs a 3D shape. More specifically, we train the model to predict a clean oriented point cloud, from which we obtain a watertight mesh using our Poisson solver and Marching Cubes. We leverage the differentiability of our Poisson solver to learn the parameters of this conditional model. Following common practice, we assume watertight meshes as ground truth and consequently supervise directly with the ground truth indicator grid obtained
from these meshes. Fig. 1 (bottom) illustrates the pipeline of our architecture for the learning-based surface reconstruction task.
Architecture: We first encode the unoriented input point cloud coordinates {c} into a feature φ. The resulting feature should encapsulate both local and global information about the input point cloud. We utilize the convolutional point encoder proposed in [52] for this purpose. Note that in the following, we will use φθ(c) to denote the features at point c, dropping the dependency of φ on the remaining points {c} for clarity. Also, we use θ to refer to network parameters in general. Given their features, we aim to estimate both offsets and normals for every input point c in the point cloud {c}. We use a shallow Multi-Layer Perceptron (MLP) fθ to predict the offset for c:
∆c = fθ(c, φθ(c)) (7)
where φ(c) is obtained from the feature volume using trilinear interpolation. We predict k offsets per input point, where k ≥ 1. We add the offsets ∆c to the input point position c and call the updated point position ĉ. Additional offsets allow us to densify the point cloud, leading to enhanced reconstruction quality. We choose k = 7 for all learning-based reconstruction experiments (see ablation study in Table 4). For each updated point ĉ, we use a second MLP gθ to predict its normal:
n̂ = gθ(ĉ, φθ(ĉ)) (8)
We use the same decoder architecture as in [52] for both fθ and gθ. The network comprises 5 layers of ResNet blocks with a hidden dimension of 32. These two networks fθ and gθ do not share weights.
Training and Inference: During training, we obtain the estimated indicator grid χ̂ from the predicted point clouds (ĉ, n̂) using our differentiable Poisson solver. Since we assume watertight and noise-free meshes for supervision, we acquire the ground truth indicator grid by running PSR on a densely sampled point clouds of the ground truth meshes with the corresponding ground truth normals. This avoids running Marching Cubes at every iteration and accelerates training. We use the Mean Square Error (MSE) loss on the predicted and ground truth indicator grid:
LDPSR = ‖χ̂− χ‖2 (9)
We implement all models in PyTorch [51] and use the Adam optimizer [30] with a learning rate of 5e-4. During inference, we use our trained model to predict normals and offsets, use DPSR to solve for the indicator grid, and run Marching Cubes [36] to extract meshes.

heading: 4 Experiments

text:
Following the exposition in the previous section, we conduct two types of experiments to evaluate our method. First, we perform single object reconstruction from unoriented point clouds. Next, we apply our method to learning-based surface reconstruction on ShapeNet [9], using noisy point clouds with or without outliers as inputs.
Datasets: We use the following datasets for optimization-based reconstruction: 1) Thingi10K [71], 2) Surface reconstruction benchmark (SRB) [65] and 3) D-FAUST [6]. Similar to prior works, we use 5 objects per dataset [19, 23, 65]. For learning-based object-level reconstruction, we consider all 13 classes of the ShapeNet [9] subset, using the train/val/test split from [11].
Baselines: In the optimization-based reconstruction setting, we compare to network-based methods IGR [19] and Point2Mesh [23], as well as Screened Poisson Surface Reconstruction4 (SPSR) [29] on plane-fitted normals. To ensure that the predicted normals are consistently oriented for SPSR, we propagate the normal orientation using the minimum spanning tree [72]. For learning-based surface reconstruction, we compare against point-based Point Set Generation Networks (PSGN) [17], patchbased AtlasNet [20], voxel-based 3D-R2N2 [11], and ConvONet [52], which has recently reported state-of-the-art results on this task. We use ConvOnet in their best-performing setting (3-plane encoders). SPSR is also used as a baseline. In addition, to evaluate the importance of our differentiable PSR optimization, we design another point-based baseline. This baseline uses the same network architecture to predict points and normals. However, instead of passing them to our Poisson solver and
4We use the official implementation https://github.com/mkazhdan/PoissonRecon.
calculate LDPSR on the indicator grid, we directly supervise the point positions with a bi-directional Chamfer distance, and an L1 Loss on the normals as done in [37]. During inference, we also feed the predicted points and normals to our PSR solver and run Marching Cubes to obtain meshes.
Metrics: We consider Chamfer Distance, Normal Consistency and F-Score with the default threshold of 1% for evaluation, and also report optimization & inference time.

heading: 4.1 Optimization-based 3D Reconstruction

text:
In this part, we investigate whether our method can be used for the single-object surface reconstruction task from unoriented point clouds or scans. We consider three different types of 3D inputs: point clouds sampled from synthetic meshes [71] with Gaussian noise, real-world scans [65], and highresolution raw scans of humans with comparably little noise [6].
Fig. 2 and Table 2 show that our method achieves superior performance compared to both classical methods and network-based approaches. Note that the objects considered in this task are challenging
due to their complex geometry, thin structures, noisy and incomplete observations. While some of the baseline methods fail completely on these challenging objects, our method achieves robust performance across all datasets.
In particular, Fig. 2 shows that IGR occasionally creates meshes in free space, as this is not penalized by its optimization objective when point clouds are unoriented. Both, Point2Mesh and our method alleviate this problem by optimizing for the Chamfer distance between the estimated mesh and the input point clouds. However, Point2Mesh requires an initial mesh as input of which the topology cannot be changed during optimization. Thus, it relies on SPSR to provide an initial mesh for objects with genus larger than 0 and suffers from inaccurate initialization [23]. Furthermore, compared to both IGR and Point2Mesh, our method converges faster.
While SPSR is even more efficient, it suffers from incorrect normal estimation on noisy input point clouds, which is a non-trivial task on its own. In contrast, our method demonstrates more robust behavior as we optimize points and normals guided by the Chamfer distance. Note that in this single object reconstruction task, our method is not able to complete large unobserved regions (e.g., the bottom of the person’s feet in Fig. 2 is unobserved and hence not completed). This limitation can be addressed using learning-based object-level reconstruction as discussed next.
To analyze whether our proposed differentiable Poisson solver is also beneficial for learning-based reconstruction, we evaluate our method on the single object reconstruction task using noise and outlieraugmented point clouds from ShapeNet as input to our method. We investigate the performance for three different noise levels: (a) Gaussian noise with zero mean and standard deviation 0.005, (b) Gaussian noise with zero mean and standard deviation 0.025, (c) 50% points have the same noise as in a) and the other 50% points are outliers uniformly sampled inside the unit cube.

heading: 4.2 Learning-based Reconstruction on ShapeNet

text:
Fig. 3 and Table 3 show our results. Compared to the baselines, our method achieves similar or better results on all three metrics. The results show that, in comparison to directly using Chamfer loss on point positions and L1 loss on point normals, our DPSR loss can produce better reconstructions in all settings as it directly supervises the indicator grid which implicitly determines the surface through the Poisson equation. SPSR fails when the noise level is high or when there are outliers in the input point cloud. We achieve significantly better performances than other representations such as point clouds, meshes, voxel grids and patches. Moreover, we find that our method is robust to strong outliers. We refer to the supplementary for more detailed visualizations on how SAP handles outliers.
Table 3 also reports the runtime for setting (a) for all GPU-accelerated methods using a single NVIDIA GTX 1080Ti GPU, averaged over all objects of the ShapeNet test set. The baselines [11, 17, 20] demonstrate fast inference time but suffer in terms of reconstruction quality while the neural implicit model [52] attains high quality reconstructions but suffers from slow inference. In contrast, our method is able to produce competitive reconstruction results at reasonably fast inference time. In addition, since ConvONet and our method share a similar reconstruction pipeline, we provide a more detailed breakdown of the runtime at a resolution of 1283 and 2563 voxels in Table 4. We use the default setup from ConvONet5. As we can see from Table 4, the difference in terms of point encoding and Marching Cubes is marginal, but we gain more than 20× speed-up over ConvONet in evaluating the indicator grid. In total, we are roughly 5× and 8× faster regarding the total inference time at a resolution of 1283 and 2563 voxels, respectively.

heading: 4.3 Ablation Study

text:
In this section, we investigate different architecture choices in the context of learning-based reconstruction. We conduct our ablation experiments on ShapeNet for the third setup (most challenging).
Number of Offsets: From Table 4 we notice that predicting more offsets per input point leads to better performance. This can be explained by the fact that with more points near the object surface, geometric details can be better preserved.
Point Cloud Encoder: Here we compare two different point encoder architectures proposed in [52]: a 2D encoder using 3 canonical planes at a resolution of 642 pixels and a 3D encoder using a feature volume with a resolution of 323 voxels. We find that the 3D encoder works best in this setting and hypothesize that this is due to the representational alignment with the 3D indicator grid.

heading: 5 Conclusion

text:
We introduce Shape-As-Points, a novel shape representation which is lightweight, interpretable and produces watertight meshes efficiently. We demonstrate its effectiveness for the task of surface reconstruction from unoriented point clouds in both optimization-based and learning-based settings. Our method is currently limited to small scenes due to the cubic memory requirements with respect to the indicator grid resolution. We believe that processing scenes in a sliding-window manner and space-adaptive data structures (e.g., octrees) will enable extending our method to larger scenes. Point cloud-based methods are broadly used in real-world applications ranging from household robots to self-driving cars, and hence share the same societal opportunities and risks as other learning-based 3D reconstruction techniques.
5To be consistent, we use the Marching Cubes implementation from [60] for both ConvONet and ours.
Acknowledgement: Andreas Geiger was supported by the ERC Starting Grant LEGO-3D (850533) and the DFG EXC number 2064/1 - project number 390727645. The authors thank the Max Planck ETH Center for Learning Systems (CLS) for supporting Songyou Peng and the International Max Planck Research School for Intelligent Systems (IMPRS-IS) for supporting Michael Niemeyer. This work was supported by an NVIDIA research gift. We thank Matthias Niessner, Thomas Funkhouser, Hugues Hopp, Yue Wang for helpful discussions in early stages of this project. We also thank Xu Chen, Christian Reiser, Rémi Pautrat for proofreading.

title: Shape As Points: A Differentiable Poisson Solver


summaries: This works suggests a novel method on how to harness the representational power of models such as CLIP. In this task the input is distorted with a known function, such as missing pixels or gaussian noise, blurring. This works proposes training a student contrastively by matching the clip representation of the clean image with student's representation of the distorted image. They propose different matching functions and explain why in their setup it will not collapse if one does not compare against every other.  The experimental setup is extensive and shows much better label efficiency when working with distorted images, they also show better noise extrapolation and better dataset tranferability in compare to baseline. Their baseline is a resnet pretrained on imagenet supervised trained on the distorted imagenet-100.

summaries: This paper proposes an inversion method that exploits a pre-trained representation for regularization. Given only corrupted images, instead of working in the pixel space, it attempts to find their matching representation pairs in the pre-trained representation space that is trained with clean images by using contrastive objective function. Assuming that one has an access to a powerful representation that is already available, this method provides a robust representation against various types and levels of distortions and train/test time discrepancy.

summaries: This paper proposes to recover the feature representation, pre-trained from CLIP, of a clean input image given a corrupted version of the image. They propose to recover the features, instead of the clean image, as their goal is to use these features for downstream tasks, e.g. classification. Furthermore, this allows for the transfer of knowledge from CLIP, which reduces the amount of labeled data needed for the downstream task. To recover the features, they train a student model via L2-Loss and contrastive loss given the clean feature from CLIP. Empirically, they evaluate on a subset of ImageNet (with only 100 classes) and consider three types of corruptions, random masking, Gaussian noise, and Gaussian blur. Empirical comparison with a baseline method, trained end-to-end with corrupted images, demonstrates their approach is more robust and requires less labeled data.

summaries: The paper consider the following problem: Let $x$ be an image, let $R$ be a feature map or representation obtained through the CLIP network, and let $A$ be a distortion process. The representation takes as input an image and yields a feature representation. The distortion process also takes an image as input, and yields a distorted image (e.g., it deletes pixels).  The paper assumes we are given the representations and corresponding distorted images, i.e., a collection $\{A(x_i), R(x_i)\}$, and the goal is to learn a student function $S(A( \cdot ))$ that is equally useful as the original representation $R(x)$ for a classification task. The paper learns the student $S$ by minimizing a contrastive loss, and the utility of the representation obtained by the student $S$ is measured by the performance on a supervised learning task, specifically on ImageNet-100.   The paper provides simulation results demonstrating that the method learns useful representations. The paper's main result is to show that the method 'outperforms a pretrained ResNet, of the same size as the robust encoder, fine-tuned end-to-end on labeled distorted images'. In addition to this result, the paper provides a number of ablations studies and compares to two reasonable baselines.

abstractText: We study a new family of inverse problems for recovering representations of corrupted data. We assume access to a pre-trained representation learning network R(x) that operates on clean images, like CLIP. The problem is to recover the representation of an image R(x), if we are only given a corrupted version A(x), for some known forward operator A. We propose a supervised inversion method that uses a contrastive objective to obtain excellent representations for highly corrupted images. Using a linear probe on our robust representations, we achieve a higher accuracy than end-to-end supervised baselines when classifying images with various types of distortions, including blurring, additive noise, and random pixel masking. We evaluate on a subset of ImageNet and observe that our method is robust to varying levels of distortion. Our method outperforms end-to-end baselines even with a fraction of the labeled data in a wide range of forward operators.

authors:
M. Asim
A. Ahmed
P. Hand

title: Invertible generative models for inverse problems: mitigating representation error and dataset bias

venue: arXiv preprint arXiv:1905.11672

authors:
A. Buades
B. Coll
J.-M. Morel

title: Non-local means denoising

venue: Image Processing On Line, 1:208–212

authors:
C. Bucila
R. Caruana
A. Niculescu-Mizil

title: Model compression

venue: KDD ’06

authors:
M. Caron
I. Misra
J. Mairal
P. Goyal
P. Bojanowski
A. Joulin

title: Unsupervised learning of visual features by contrasting cluster assignments

venue: arXiv preprint arXiv:2006.09882

authors:
T. Chen
L. Li

title: Intriguing properties of contrastive losses

authors:
T. Chen
S. Kornblith
M. Norouzi
G.E. Hinton

title: A simple framework for contrastive learning of visual representations

authors:
T. Chen
S. Kornblith
K. Swersky
M. Norouzi
G.E. Hinton

title: Big self-supervised models are strong semi-supervised learners

authors:
A. Coates
A. Ng
H. Lee

title: An analysis of single-layer networks in unsupervised feature learning

venue: Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics,

authors:
J.P. Cohen
P. Morrison
L. Dao

title: Covid-19 image data collection

authors:
M.Z. Darestani
A. Chaudhari
R. Heckel

title: Measuring robustness in deep learning based compressive sensing

venue: arXiv preprint arXiv:2102.06103

authors:
M. Dhar
A. Grover
S. Ermon

title: Modeling sparse deviations for compressed sensing using generative models

venue: arXiv preprint arXiv:1807.01442

authors:
S. Dodge
L. Karam

title: A study and comparison of human and deep learning recognition performance under visual distortions

venue: 2017 26th International Conference on Computer Communication and Networks (ICCCN), pages 1–7

authors:
F.L. Gómez
A. Eftekhari
V. Cevher

title: Fast and provable admm for learning with generative priors

venue: arXiv preprint arXiv:1907.03343

authors:
M. Gutmann
A. Hyvärinen

title: Noise-contrastive estimation: A new estimation principle for unnormalized statistical models

venue: Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics,

authors:
K. He
H. Fan
Y. Wu
S. Xie
R. Girshick

title: Momentum contrast for unsupervised visual representation learning

venue: In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR),

authors:
R. Heckel
P. Hand

title: Deep decoder: Concise image representations from untrained nonconvolutional networks

venue: arXiv preprint arXiv:1810.03982

title: Data-efficient image recognition with contrastive predictive coding

venue: Proceedings of the 37th International Conference on Machine Learning,

authors:
D. Hendrycks
T. Dietterich

title: Benchmarking neural network robustness to common corruptions and perturbations

venue: International Conference on Learning Representations

authors:
G. Hinton
O. Vinyals
J. Dean

title: Distilling the knowledge in a neural network

venue: NIPS Deep Learning and Representation Learning Workshop

authors:
C.-H. Ho
N. Vasconcelos

title: Contrastive learning with adversarial examples

authors:
H. Hosseini
B. Xiao
R. Poovendran

title: Google’s cloud vision api is not robust to noise

venue: 2017 16th IEEE International Conference on Machine Learning and Applications (ICMLA), pages 101–105

authors:
Z. Jiang
T. Chen
T. Chen
Z. Wang

title: Robust pre-training by adversarial contrastive learning

venue: NeurIPS

authors:
M. Kabkab
P. Samangouei
R. Chellappa

title: Task-aware compressed sensing with generative adversarial networks

venue: Thirty-Second AAAI Conference on Artificial Intelligence

authors:
P. Khosla
P. Teterwak
C. Wang
A. Sarna
Y. Tian
P. Isola
A. Maschinot
C. Liu
D. Krishnan

title: Supervised contrastive learning

authors:
M. Kim
J. Tack
S.J. Hwang

title: Adversarial Self-Supervised contrastive learning

authors:
D.P. Kingma
J. Ba

title: Adam: A method for stochastic optimization

authors:
K. Kotar
G. Ilharco
L. Schmidt
K. Ehsani
R. Mottaghi

title: Contrasting contrastive Self- Supervised representation learning models

title: Learning multiple layers of features from tiny images

title: Pseudo-label : The simple and efficient semi-supervised learning method for deep neural networks

authors:
M. Lustig
D. Donoho
J.M. Pauly

title: Sparse mri: The application of compressed sensing for rapid mr imaging

venue: Magnetic Resonance in Medicine: An Official Journal of the International Society for Magnetic Resonance in Medicine, 58(6):1182–1195

authors:
G. Ongie
A. Jalal
C.A. Metzler
R.G. Baraniuk
A.G. Dimakis
R. Willett

title: Deep learning techniques for inverse problems in imaging

venue: IEEE Journal on Selected Areas in Information Theory, 1(1):39–56

authors:
A. Oord
Y. Li
O. Vinyals

title: Representation learning with contrastive predictive coding

authors:
P. Pandit
M. Sahraee-Ardakan
S. Rangan
P. Schniter
A.K. Fletcher

title: Inference with deep generative priors in high dimensions

venue: arXiv preprint arXiv:1911.03409

authors:
A. Paszke
S. Gross
F. Massa
A. Lerer
J. Bradbury
G. Chanan
T. Killeen
Z. Lin
N. Gimelshein
L. Antiga
A. Desmaison
A. Kopf
E. Yang
Z. DeVito
M. Raison
A. Tejani
S. Chilamkurthy
B. Steiner
L. Fang
J. Bai
S. Chintala

title: Pytorch: An imperative style

venue: highperformance deep learning library. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché- Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems 32, pages 8024–8035. Curran Associates, Inc.

authors:
A. Radford
J.W. Kim
C. Hallacy
A. Ramesh
G. Goh
S. Agarwal
G. Sastry
A. Askell
P. Mishkin
J. Clark
G. Krüger
I. Sutskever

title: Learning transferable visual models from natural language supervision

authors:
O. Russakovsky
J. Deng
H. Su
J. Krause
S. Satheesh
S. Ma
Z. Huang
A. Karpathy
A. Khosla
M.S. Bernstein
A. Berg
L. Fei-Fei

title: Imagenet large scale visual recognition challenge

venue: International Journal of Computer Vision, 115:211–252

authors:
J. Scarlett
V. Cevher

title: Limits on support recovery with probabilistic models: An informationtheoretic framework

venue: IEEE Transactions on Information Theory, 63(1):593–620

authors:
D. Temel
J. Lee
G. Alregib

title: Cure-or: Challenging unreal and real environments for object recognition

venue: 2018 17th IEEE International Conference on Machine Learning and Applications (ICMLA), pages 137–144

authors:
Y. Tian
D. Krishnan
P. Isola

title: Contrastive multiview coding

venue: arXiv preprint arXiv:1906.05849

title: Contrastive representation distillation

authors:
Y. Tian
L. Yu
X. Chen
S. Ganguli

title: Understanding self-supervised learning with dual deep networks

authors:
T. Wang
P. Isola

title: Understanding contrastive representation learning through alignment and uniformity on the hypersphere

venue: ICML

authors:
Z. Wu
Y. Xiong
S.X. Yu
D. Lin

title: Unsupervised feature learning via non-parametric instance discrimination

venue: In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR),

authors:
Q. Xie
E. Hovy
M.-T. Luong
Q.V. Le

title: Self-training with noisy student improves imagenet classification

venue: 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 10684–10695

authors:
Y. Zhou
S. Song
N.-M. Cheung

title: On classification of distorted images with deep convolutional neural networks

venue: 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1213–1217

heading: 1 Introduction

text:
Modern representation learning networks like CLIP [35] are showing incredible performance for image classification, even for zero-shot problems with labels not seen during training. Training these encoders comes at a staggering cost and requires datasets and computing resources only available to very few organizations. In this paper we show how to leverage this pretrained power for a new family of problems in the presence of image corruptions or other types of measurements.
Inverse problems involve reconstructing an unknown vector x from measurements y = A(x). Typically, the forward operator A corrupts the unknown vector x and reduces its dimension, i.e. the observations y live in a lower-dimensional space compared to x. In the special case of linear inverse problems, the forward operator is simply a matrix and the measurements are in the form y = Ax + noise. Special cases of linear inverse problems include image denoising, inpainting, super-resolution, compressed sensing used in medical tomography, seismic geological imaging and many others, see e.g. [31] for a recent overview.
∗Equal contribution. Code available at https://github.com/Sriram-Ravula/Contrastive-Inversion.
35th Conference on Neural Information Processing Systems (NeurIPS 2021).
In this paper, we introduce the study of a new family of inverse problems: reconstructing the representation of an image given a corrupted or measured input. Formally, if a (clean) image is x and its CLIP representation is R(x), we would like to obtain that representation by only observing a highly corrupted input A(x). This is impossible if the forward process A removes information needed to obtain the representation. Surprisingly, we show that we can recover representations that are useful for downstream tasks, even from extremely corrupted versions of the image.
We introduce a robust encoder S that is trained to imitate the behavior of the pretrained CLIP encoder acting on clean images x. However, the input to the robust encoder is only corrupted images A(x) that are created by applying the forward operator on x. Our approach is illustrated in Figure 2. The teacher encoder is the pretrained CLIP, and the student encoder is our robust encoder operating on corrupted images. Formally, the robust encoder S(A(x)) is trained to approximate R(x) using a contrastive loss.
Many applications, such as object recognition with low-cost cameras, remote sensing and aerial imaging rely on noisy or blurry data and can face occlusions or sensor corruptions. As we demonstrate in our experiments in the Appendix, normal CLIP fails on highly corrupted images. Our procedure allows us to transfer the power of CLIP to heavily corrupted images in downstream tasks, with relatively little extra training.

heading: 1.1 Results

text:
We show that our method is able to obtain useful representations even under extreme corruptions such as removing 90% of the pixels as shown in the top panel of Figure 1. The highly corrupted images enter our robust encoder and the obtained representation is used in a linear classifier to produce ImageNet-100 labels. Our main result is that our method outperforms a pretrained ResNet (of the same size as our robust encoder) fine-tuned end-to-end on labeled distorted images.
Using less labeled data For some corruption levels, we are able to outperform end-to-end fine-tuned ResNets using as little as 10% of labeled samples. This is even when the fine-tuned baseline uses 100% of ImageNet-100 labels for training. The primary advantage of our model is that our robust encoder observes representations from the pretrained CLIP encoder, which was trained with a much bigger dataset compared to ImageNet. Still, the fact that this implicit advantage in the representations and 10% of labeled data is sufficient to outperform a supervised trained ResNet fine-tuned with 10 times more labeled data is very surprising and illustrates the power and versatility of pretrained representation learners.
Robustness to noise and data shifts Our method is very robust to changes in the forward operators, data statistics and label shifts. We experiment with three classes of forward operators: random pixel masking, additive Gaussian noise, and Gaussian blurring distortions. For each, we train and test on a wide-range of distortion levels, from slight to severe corruption. We show that our robust encoder produces useful representations even when the level of corruption is outside its training domain.
We show that our representations are useful for a wide range of tasks, without requiring knowledge of the task when the robust encoder is trained. We illustrate excellent classification accuracy across five datasets, frequently outperforming end-to-end supervised baselines trained with knowledge of the target task. Our experiments include a chest X-ray COVID pneumonia task which has very different morphology compared to ImageNet. Surprisingly, the same universal representations, combined with a custom linear probe are very successful across all tasks.
Contrastive versus MSE training We formulate the contrastive student training method as a regularization upon the simple mean squared error loss between student embeddings of a distorted image and teacher embeddings of the clean version of that image. We analyze the effects of this regularization on the training dynamics. Our results empirically show that simple MSE is worse in most cases, further strengthening our argument for the usefulness of contrastive learning in this setting.

heading: 2 Related Work

text:
Robust Image Recognition It has been shown that classifiers that perform well on clean image tasks are not robust to common image distortions [21]. Several datasets have been proposed specifically to benchmark generalization of classifier performance to natural distortions [18, 38]. Related to our work, [45] and [12] fine-tune pretrained classifiers on distorted data, which seems to yield better performance than end-to-end training. However even under these training processes, modern classifiers exhibit inferior performance to human vision on distorted data [12].
Inverse problems There is significant recent literature on solving inverse problems including denoising, inpainting and deconvolution for deblurring. While classical techniques rely on sparsity based priors e.g. [30, 37], recent techniques include data-driven deep-learning methods [1, 10, 13, 33] as well as combinations of sparsity and generative methods [11] and untrained deep nets [16]. Our work focuses on recovering the representation of an image, as opposed to an image itself, so it is also related to task-aware sensing [23]. However, our approach is fundamentally different from all
previous inverse problems since instead of trying to reconstruct the image x, we aim to reconstruct a representation R(x), which lives in a different space. The other important distinction is that even though our corruption processes are linear in the pixel space, they are non-linear with respect to the representation vector we try to recover. In effect, we are solving a non-linear inverse problem in a supervised way using a contrastive loss. In the Appendix, we compare to methods that attempt to solve the inverse problem in pixel space, then apply a classifier on the recovered image.
Contrastive Representation Learning Self-supervised representation learning has recently exploded in popularity, largely due to the success of constrastive losses in learning representations from unlabeled data [4, 6, 7, 15, 17, 43]. These techniques are able to generate highly general embeddings of images that are effective for many types of downstream tasks, even on domains that were not explicitly considered in training [27]. Contrastive losses generally operate based on a simple push-pull principle: images desired to be close in embedding space are pushed together, while unrelated images are pulled apart. One particularly popular choice of contrastive loss is the InfoNCE loss, derived from techniques for noise contrastive estimation [14], and popularized in the self-supervised setting in [32]. Several works have considered adversarial training frameworks to yield representations that are more robust to adversarial attacks [20, 22, 25]. However these works only consider adversarial robustness and not robustness to common corruptions. Our approach is similar to that of [24], which employs a variant of the InfoNCE loss for a supervised setting. This differs from our work in that we exclusively focus on robustness to natural corruptions. Moreover, our contrastive training step can be performed without any task-specific labeled data. This is made possible by the use of the powerful embeddings provided by CLIP, and allows the embeddings to be used for downstream tasks on multiple datasets.
Knowledge Transfer Methods Our work is closely related to prior works which aim to distill, reduce, or transfer the knowledge from one network to another for a specific task [3, 19, 29, 44]. Of note is [40], where the authors use a contrastive objective to transfer representations from a teacher network to a student network. Our work diverges in that we do not transfer from a larger, more powerful teacher to a smaller student, but rather transfer between a teacher and student of the same architecture initialized from the same weights. In addition, although the authors test on cross-modal transfer tasks such as transferring between color channels, transferring representations between clean and distorted images is a different task: we try to extract the same high-level information from less data as opposed to different, but related data.

heading: 3 Method

text:
Our problem is to recover the embeddings of clean images when we only have access to highly corrupted versions of the images. The encoder R(·) is assumed to yield high-quality representations for a variety of domains. The distortion process A(·) is assumed to be a known forward operator that greatly distorts images. We assume A(·) is sufficiently severe as to inhibit the performance of the encoder R(·), but not so severe that recovery is impossible. From a collection of input images {xi}Ni=1, we are only given access to the distorted images and the representations of the clean images {A(xi), R(xi)}Ni=1. Our approach is to learn a student function S so that S(A(·)) is equally useful as the teacher representation R(·). We measure the utility of a representation by the performance on an unspecified downstream supervised learning task.
Least squares Loss One potential approach for the task of recovering the teacher’s embeddings from the corrupted inputs is to minimize the expected `2 distance in embedding space between the clean teacher embeddings R(·) and the predicted student representations S(A(·)). If both R(·) and S(·) are constrained to have `2-normalized outputs, then the empirical least squares loss becomes:
L̂MSE(S;R,A) := −1 N N∑ i=1 〈S(A(xi)), R(xi)〉, (1)
where we have expanded and subtracted out constant terms. However, this process may not yield the most effective embedding of the corrupted data. Due to either limitations in the training process, the severity of the distortion process, or the generalization properties of the approximate minimizers of L̂MSE, the learned embeddings S(A(·)) may not be as useful for downstream tasks as embeddings learned using the other losses we consider.
Contrastive Loss Inspired by recent advances in self-supervised representation learning, we learn S by minimizing a contrastive loss. We consider the following variant on the popular InfoNCE loss:
L̂contr(S; τ,R,A) := −1 N N∑ i=1 log exp(K(i, i)/τ)∑N j=1 exp(K(i, j)/τ) , (2)
where K(i, j) := 〈S(A(xi)), R(xj)〉 measures the similarity between the learned embedding of A(xi) and the clean embedding of xj , and τ is a temperature hyperparameter. We follow [5, 42] and rewrite L̂contr in terms of explicit ‘pull‘ and ‘push‘ terms as :
L̂contr(S; τ,R,A) = 1 τ L̂MSE(S;R,A) + L̂unif(S; τ,R,A), (3)
where the second term, referred to as the uniformity term, is defined as:
L̂unif(S; τ,R,A) := 1 N N∑ i=1 log N∑ j=1 eK(i,j)/τ . (4)
The first term of L̂contr is simply L̂MSE which encourages alignment of S(A(xi)) with R(xi), and the L̂unif term encourages the learned representations for corrupted data to be dissimilar from all other representations of clean data.
The primary difference between Equation 3 and the InfoNCE loss commonly used in self-supervised learning is the choice of the similarity measureK(·, ·). Without access to specified target embeddings, K(i, j) is typically chosen to be the inner product between projections of the embeddings of xi and xj , (or between xi and a positive example in the case of K(i, i)). Here, the uniformity term is necessary to prevent representation collapse, where in our setting L̂MSE alone suffices to prevent this degenerate case. We note that alternative choices for K(·, ·) in L̂unif may be employed. For example, we could also contrast the student embeddings S(A(·)) across two different images. We ablate against other choices in the experimental section and find that our choice of K(·, ·) is one of several that exhibits comparable performance.
Effect of uniformity term To examine the effect of the uniformity term of the loss on the training dynamics, we consider the gradients of L̂contr with respect to the parameters of the encoder S. We first decompose L̂unif to consider the contributions of each individual data point:
L̂unif(S; τ,R,A) = 1 N N∑ i=1 L̂unifi (S; τ,R,A), L̂unifi (S; τ,R,A) := log N∑ j=1 e K(i,j) τ .
Then the gradient of L̂contr with respect to the parameters of S may be written as
∇SL̂contr(S; τ,R,A) = −1 τ ∇SL̂MSE(S;R,A) + 1 τN N∑ i=1 N∑ j=1 wi(j)∇SK(i, j),
where wi(j) := exp ( K(i, j)/τ − L̂unifi (S;T,R,A) ) .
This can be interpreted as follows. The weighting of the first term by τ−1 balances the gradients ∇SK(i, i) and ∑ j ∇SK(i, j), as is common with other choices of contrastive losses (c.f. Theorem 2 in [41]). Noting that for each i, the weights wi(j) sum to 1, the gradient of each individual uniformity term L̂unifi is a convex combination of the gradients of the similarity terms K(i, ·). The individual similarity terms are weighted exponentially proportionally to each K(i, j)’s contribution to the total uniformity loss. As τ decreases, the weights place greater emphasis on terms that are most similar. As τ is commonly chosen to be less than 1, the dynamics automatically reflect the influence of the ‘hardest’ negative examples.
When is perfect recovery possible? Finally, we describe conditions which allow for perfect recovery of the clean embeddings from the corrupted images in the training set. The first condition is that the corruption process should not be too destructive: it suffices to assume that the implication A(xi) = A(xj) =⇒ R(xi) = R(xj) holds for all xi, xj in the training set, i.e., there exists a function which attains exact recovery. In this case, any S∗ that minimizes L̂MSE has S∗(A(xi)) = R(xi) for all xi in the training set. To argue for the same recovery guarantees when optimizing L̂contr, we need to assume that the teacher R(·) provides a well-separated embedding of the training data. Proposition 1. If R is a minimizer of the uniformity term
R ∈ argmin f N∑ i=1 log N∑ j=1 exp 〈f(xi), f(xj)〉 τ ,
then any encoder S∗ ∈ argminS L̂contr(S; τ,R,A) exactly recovers the target embedding, S∗(A(xi)) = R(xi) for all xi in the training set.
Training procedure Our final method consists of (1) the contrastive step, in which the student learns representations for the distorted images and (2) a fine-tuning step, where we train a linear classifier on top of the learned representations. During the second step, the student encoder is kept frozen.

heading: 4 Experiments

text:
We show that the proposed training process recovers useful representations from corrupted inputs for a variety of forward operators, evaluating several different classification tasks. We start by evaluating the representation quality when the distortion process and data distribution are the same during both training and inference, examining the label-efficiency of our approach.
Then we evaluate our approach when the severity of the distortions changes at test time, when the test data distribution differs from the one used during training, and when the labels are shifted. Finally, we ablate against alternative formulations of our loss function.
For all experiments, we perform contrastive training for the robust encoder using a 100-class subset of ImageNet, which we refer to as ImageNet-100, [36, 39] to reduce computational resources. Our target representations are attained from the CLIP ResNet-101. We find robust representations with contrastive learning and evaluate by training a linear classifier on top of the frozen representations. In all experiments, the distortions are applied randomly to each batch of images, and independently for each image in the batch. Where applicable, we report our results over 10 different instances of random corruptions applied on the evaluation images. Our baselines are built on a ResNet-101 initialized with weights from supervised training on the full ImageNet dataset. The final fully-connected layer is replaced with a 100-dimensional output. We fine-tune the whole model in a supervised fashion using distorted inputs and their correct labels from ImageNet-100. The baseline is trained for 25 epochs with a batch size of 64. Our robust encoder is trained for 25 epochs with a batch size of 256, and the linear probe on top of it is trained for 10 epochs. Additional experiments, as well as further details on training and hyperparameter choices are discussed in the appendix and in our provided code.

heading: 4.1 Known Data Distribution and Forward Operator

text:
In this section, we evaluate the quality of the learned robust representations for classifying images from the validation set of ImageNet-100, using the same distortions during training and inference. These experiments demonstrate the usefulness of our method for vision inverse problems where the data distribution and forward operator are both known at training time. We also demonstrate the label-efficiency of our approach by evaluating our learned representation when only few labeled samples are available to train the linear classifier.
Setup We train the robust contrastive model and the baseline as described above. Eight different distortion processes are examined: Gaussian blur with (kernel size, standard deviation) of (21, 5) and (37, 9); additive Gaussian noise with standard deviations 0.1, 0.3, and 0.5; and random pixel mask with 50%, 75%, and 90% of the pixels missing. We evaluate our method against the baseline in top-1
0.1% 0.5% 1% 5% 10% 100%
Fraction of Labels
45
50
55
60
65
70
75
80
85
T op -1
A cc ur ac y (%
)
Random mask, ours Random mask, baseline Gaussian noise, ours Gaussian noise, baseline
Figure 3: Accuracies using a varying fraction of labeled samples to train a linear probe. We train robust encoders on images with 90% random pixel masking and additive Gaussian noise with standard deviation 0.5, and fit a linear classifier on the learned representations using varying fractions of labeled training samples. We compare to a supervised baseline that uses all of the labeled training samples. Results are averaged over 10 random instantiations of corruptions on the ImageNet-100 validation dataset. We omit error bars as standard error is insignificant.
accuracy on the validation set of ImageNet-100, under the same distortion used to train each model. To demonstrate the label-efficiency of our method, we also train a linear probe with only 10% of the labeled data, and for two of our models we train linear probes using various amounts of labeled data.
Results In Table 1, we see that training a linear probe on top of the representations learned by our procedure greatly improves accuracy compared to the supervised baseline. This further solidifies our original motivation. Furthermore, we can see that using only 10% of labeled samples is sufficient for our model to outperform the baseline in most cases. More fine-grained label-efficiency results can be seen in Figure 3, where we show that in two of these cases, using just 5% of the labeled data makes our model outperform or be competitive with the baseline trained using all the labels.

heading: 4.2 Known Data Distribution and Unknown Forward Operator

text:
In many settings, the data distribution and the type of distortion will be known at training time, but the severity of the distortion will be unknown. We consider the case where the severity of the distortion at test-time is greater than what was seen during training.
Setup First, we train a model using training images with between 50 and 95% of pixels randomly masked. In addition, we train a model using images with additive Gaussian noise with random standard deviation between 0.1 and 0.3. Once fully trained, we fit a linear classifier on top of the learned representations for each model using the same distortions. We also train two supervised baselines end-to-end with the same distortions. We evaluate the models trained with pixel masking on images with a fixed level of 96 to 99% percent missing pixels, and the networks trained with noise on images with additive Gaussian noise using a fixed standard deviation between 0.35 and 0.5.
Results In Figure 4, we see that with an increase in noise levels, the accuracy of the models does decrease. However, the linear probe trained on the entire dataset achieves better results than the baseline end-to-end supervised model. This shows that our model is more robust, even when the distortions are greater than those expected during training.

heading: 4.3 Unknown Data Distribution and Known Forward Operator

text:
In this section we evaluate how well ImageNet pretraining with distortions allows the learned representations to transfer to different datasets. We use the same forward operator during training and inference to isolate the quality of the embeddings learned during the contrastive step even in the presence of distortion.
Setup Five datasets are chosen to evaluate transferability of robust representations. (1) CIFAR-10 and (2) CIFAR-100 [28], and (3) STL-10 [8]. (4) The COVID-19 Chest X-ray dataset [9], (5) We generate another random 100-class subset of ImageNet [36] from the remaining 900 classes we did not use for ImageNet-100, which we refer to as ImageNet-100B.
We train the same models under the same distortions as outlined in Section 4.2, then fit linear classifiers for the new datasets on top of the fixed representations. Preprocessing details for each dataset may be found in the Appendix. We calculate top-1 accuracy of each model for classifying distorted images from the validation or test set of each of the new datasets. The images are distorted using the same forward operators used to train the networks.
Results Table 2 shows that our approach can achieve good results in a variety of datasets. In CIFAR-10 and CIFAR-100, we get results comparable to the baseline. In STL-10, we have greater top 1 accuracy for both noise settings. For the COVID X-ray dataset, we get mixed results, where we beat the baseline for the random masking model, but we lose for the additive Gaussian noise model. The most surprising result is the vast increase in accuracy for the alternative ImageNet-100B dataset.
This shows that due to the supervised training of the baseline, some information from the labels leaks into the representation, leading to worse performance on a related, but ultimately different dataset.

heading: 4.4 Label Shift

text:
An important factor in determining the robustness of a model is how gracefully it fails in the presence of unseen data modalities at inference time. For instance, if a network trained to distinguish dogs from cars is shown an image of a cat, the network should produce a embedding which is more likely to be classified as a dog than a car. This is even more important for inverse problems: a single distorted image may result from the same forward operator being applied to any number of original images. If a distorted image from a class outside of the training classes makes a network output an unexpected or uninformative representation, then the network is likely also brittle to shifts in the data distribution within the classes it knows at inference time. We evaluate the quality of representations produced by our method for distorted images from classes outside of the training dataset.
Setup We use the same models described in Section 4.2. To simulate images from unseen classes, we identify 5 classes from the validation set of full ImageNet that were not used in our training data, but are similar to classes within our training data. We replace the labels of these new images with the label of the classes in our training data that are similar to them, as seen in Table 3. Evaluation is done on images from these replacement classes, under fixed levels of distortion, with 50% to 99% missing pixels on the random masking model and 0.05 to 0.5 standard deviation for the Gaussian noise model.
Results In Figure 5, we can see that, for the random pixel mask case, we consistently outperform the baseline across all noise levels. For the Gaussian noise case, our model has slightly lower top-1 accuracy for small noises. This can be explained by the fact that small distortions do not drastically alter the image, so the baseline is bolstered by its ImageNet pretraining. However, as the noise level increases, the baseline model degrades much more quickly.
90% 92% 94% 96% 98% 100%
Fraction of Missing Pixels
0
10
20
30
40
50
60
70
80
T op -1
A cc ur ac y (%
)
Student v. Teacher Student v. Student Student v. Both NT-Xent MSE
Figure 6: Ablation study. We evaluate the model trained on 50% to 95% random masking of pixels, using several variants of the uniformity term in the contrastive loss. Results are averaged over 10 random instantiations of corruptions on the ImageNet-100 validation dataset. We omit error bars as standard error is insignificant.

heading: 4.5 Ablation: Training Loss

text:
Finally, we ablate against various choices for the uniformity term in the contrastive loss. We consider variants of L̂unif where we compare representations of i) the student and teacher, ii) the student and the student, iii) a sum of both of the above. We also consider L̂MSE, i.e. no uniformity term, as well as one variant where every representation is contrasted with every other representation (irrespective of student or teacher), which we note is the same formulation as NT-Xent as used in SimCLR [6].
Setup We use the same random masking model described in Section 4.2, with a similar evaluation on fixed masking levels of 90-99%. Explicit formulations of each of the losses is in the appendix.
Results We draw two main conclusions from Figure 6. First, the MSE loss performs worst, indicating the benefit of the uniformity term in the loss. Even though we have access to the pretrained representations, it is not simple for the model to exploit the encoded information. If this were the case, MSE would be as effective as our contrastive loss. Second, we see that all student comparisons perform roughly equivalently, but are clearly more effective than MSE and NT-Xent.

heading: 5 Conclusion

text:
In this work, we propose a method for training image representation networks which are robust to various distortions on the input data. Our method has potential for improving the practical applications of powerful pre-trained models. Indeed, images in real-world settings are rarely exemplary: every stage of the imaging process, from capture to storage to transmission and display, can introduce noise or distortions in the images. Moreover, our process helps reduce the cost of training such models, both with respect to computation since we can add robustness to a pre-trained model instead of training one from scratch, as well as with respect to label efficiency since our method mainly relies on large amounts of inexpensive unlabeled data.
Several significant open problems emerge from this work. As we can see in Appendix D.8, our method requires some prior knowledge of the type of distortions that will occur to the image. This is due to the fact that our method relies on training the student to match the representations of the teacher, which is significantly more difficult when the type of corruption (and thus, the representation itself) changes drastically. As future work, we aim to extend our method beyond this limitation, possibly by finetuning the teacher as well as the student, to produce representations that are more easily matched under different types of distortions. In addition, another interesting research direction is how well this method performs under adversarially-chosen inputs.

heading: Acknowledgements

text: This research has been supported by NSF Grants CCF 1763702, 1934932, AF 1901281, 2008710, 2019844 the NSF IFML 2019844 award as well as research gifts by Western Digital, Interdigital, WNCG and MLL, computing resources from TACC and the Archie Straiton Fellowship.

title: Inverse Problems Leveraging Pre-trained Contrastive Representations