summaries: The submission presents a way of improving training of deep implicit networks, that take Cartesian coordinates (e.g., 2D or 3D) as inputs. The method uses additional embeddings of the input coordinates, similarly to Positional Encodings and variations (SIREN, FFN), with 2 differences: - embeddings corresponding to higher spatial frequencies are only be made available later during the optimization phase, and only if the network does not fit well enough - they can be made available selectively on different parts of the input space, based on a spatial grid. This method makes it possible to better represent inputs where the maximal spatial frequency changes a lot between regions (e.g., an image with large smooth regions, and regions of higher detail), and compares favorably to similar methods. This paper proposes an improvement of implicit function approximation for image and mesh generation. The main contribution is to use a progressive low resolution to high resolution optimization as well as spatially adapt this `masking`. Performance is shown to be better than FFN by a significant margin. This paper highlights a problem with Position Encoding models that use Fourier features: they're very sensitive to a bandwidth parameter that trades-off between reproduction of low- and high-spatial frequency features. Their solution is an iterative feedback algorithm that reconstructs images in a coarse-to-fine manner. They describe incremental improvements across-the-board, and large and impressive improvements in their 2D silhouette reconstruction task. Implicit neural representations are a promising approach to learning compressed representations of data --- for example they can learn to represent an image where the network inputs are the (x,y) coordinates and the outputs are the RGB pixel values. This idea has all sorts of applications in data science and physics.   This work provides a method for improving the optimization of these implicit neural representations. It addresses the challenge that standard MLPs are biased towards learning low-spatial frequencies, and that a dataset (e.g. an image to compress) can have different frequency distributions in different regions.  The paper presents a technique for dealing with the problem of choosing bandwidths for positional encodings in implicit functions. The method presented uses higher frequency positional encodings as training progresses and clips the maximum frequency when a quality criterion has been reached for a position (therefore it is spatially adaptive) in order to avoid the appearance of artifacts.  Results on several domains (images, 3d occupancy, and mesh deformations) are presented with good results in all cases.
abstractText: Multilayer-perceptrons (MLP) are known to struggle with learning functions of high-frequencies, and in particular cases with wide frequency bands. We present a spatially adaptive progressive encoding (SAPE) scheme for input signals of MLP networks, which enables them to better fit a wide range of frequencies without sacrificing training stability or requiring any domain specific preprocessing. SAPE gradually unmasks signal components with increasing frequencies as a function of time and space. The progressive exposure of frequencies is monitored by a feedback loop throughout the neural optimization process, allowing changes to propagate at different rates among local spatial portions of the signal space. We demonstrate the advantage of SAPE on a variety of domains and applications, including regression of low dimensional signals and images, representation learning of occupancy networks, and a geometric task of mesh transfer between 3D shapes.
authors.name: Amir Hertz
authors.name: Olga Sorkine-Hornung
authors.name: Daniel Cohen-Or
references.authors: Matan Atzmon Yaron Lipman
references.title: SAL: sign agnostic learning of shapes from raw data
references.venue: IEEE/CVF Conference on Computer Vision and Pattern Recognition,
references.authors: Oshrat Bar Amnon Drory Raja Giryes
references.title: A spectral perspective of neural networks robustness to label noise, 2021
references.authors: Alberto Bietti Julien Mairal
references.title: On the inductive bias of neural tangent kernels
references.venue: In Advances in Neural Information Processing Systems,
references.authors: Yuan Cao Zhiying Fang Yue Wu Ding-Xuan Zhou Quanquan Gu
references.title: Towards understanding the spectral bias of deep learning
references.venue: In International Joint Conference on Artificial Intelligence,
references.authors: Rohan Chabra Jan E Lenssen Eddy Ilg Tanner Schmidt Julian Straub Steven Lovegrove Richard Newcombe
references.title: Deep local shapes: Learning local sdf priors for detailed 3d reconstruction
references.venue: In European Conference on Computer Vision,
references.authors: Zhiqin Chen Hao Zhang
references.title: Learning implicit fields for generative shape modeling
references.venue: In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR),
references.authors: Ch Dash Ajit Kumar Behera Satchidananda Dehuri Sung-Bae Cho
references.title: Radial basis function neural networks: A topical state-of-the-art survey
references.venue: Open Computer Science,
references.authors: Thomas Davies Derek Nowrouzezahrai Alec Jacobson
references.title: On the effectiveness of weightencoded neural implicit 3d shapes
references.authors: Zhantao Deng Jan Bednařík Mathieu Salzmann Pascal Fua
references.title: Better patch stitching for parametric surface reconstruction
references.venue: arXiv preprint arXiv:2010.07021,
references.authors: Benjamin Eckart Kihwan Kim Alejandro Troccoli Alonzo Kelly Jan Kautz
references.title: Accelerated generative models for 3d point cloud data
references.venue: In Proceedings of the IEEE conference on computer vision and pattern recognition,
references.authors: Philipp Erler Paul Guerrero Stefan Ohrhallinger Niloy J Mitra Michael Wimmer
references.title: Points2surf learning implicit surfaces from point clouds
references.venue: In European Conference on Computer Vision,
references.authors: D. Ezuz B. Heeren O. Azencot M. Rumpf M. Ben-Chen
references.title: Elastic correspondence between triangle meshes
references.venue: Computer Graphics Forum,
references.authors: Kyle Genova Forrester Cole Daniel Vlasic Aaron Sarna William T. Freeman Thomas Funkhouser
references.title: Learning shape templates with structured implicit functions
references.venue: In International Conference on Computer Vision (ICCV),
references.authors: Kyle Genova Forrester Cole Avneesh Sud Aaron Sarna Thomas Funkhouser
references.title: Local deep implicit functions for 3d shape
references.venue: In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,
references.authors: Amos Gropp Lior Yariv Niv Haim Matan Atzmon Yaron Lipman
references.title: Implicit geometric regularization for learning shapes
references.venue: In International Conference on Machine Learning,
references.authors: Zekun Hao Hadar Averbuch-Elor Noah Snavely Serge Belongie
references.title: Dualsdf: Semantic shape manipulation using a two-level representation
references.venue: In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,
references.authors: Alec Jacobson Ilya Baran Jovan Popovic Olga Sorkine
references.title: Bounded biharmonic weights for real-time deformation
references.venue: ACM Trans. Graph.,
references.authors: Chiyu "Max" Jiang Avneesh Sud Ameesh Makadia Jingwei Huang Matthias Niessner Thomas Funkhouser
references.title: Local implicit grid representations for 3d scenes
references.venue: In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR),
references.authors: Guolin Ke Di He Tie-Yan Liu
references.title: Rethinking positional encoding in language pre-training
references.venue: In International Conference on Learning Representations,
references.authors: L.J. Latecki R. Lakamper
references.title: Shape similarity measure based on correspondence of visual parts
references.venue: IEEE Transactions on Pattern Analysis and Machine Intelligence,
references.authors: Chen-Hsuan Lin Wei-Chiu Ma Antonio Torralba Simon Lucey
references.title: Barf: Bundle-adjusting neural radiance fields
references.venue: arXiv preprint arXiv:2104.06405,
references.authors: Shichen Liu Shunsuke Saito Weikai Chen Hao Li
references.title: Learning to infer implicit surfaces without 3d supervision
references.venue: arXiv preprint arXiv:1911.00767,
references.authors: Xuanqing Liu Hsiang-Fu Yu I. Dhillon Cho-Jui Hsieh
references.title: Learning to encode position for transformer with continuous dynamical model
references.venue: In ICML,
references.authors: Antoine Liutkus Ondřej Cífka Shih-Lun Wu Umut Şimşekli Yi-Hsuan Yang Gaël Richard
references.title: Relative positional encoding for transformers with linear complexity, 2021
references.authors: Julien N.P. Martel David B. Lindell Connor Z. Lin Eric R. Chan Marco Monteiro Gordon Wetzstein
references.title: Acorn: Adaptive coordinate networks for neural representation
references.venue: ACM Trans. Graph. (SIGGRAPH),
references.authors: Ricardo Martin-Brualla Noha Radwan Mehdi S.M. Sajjadi Jonathan T. Barron Alexey Dosovitskiy Daniel Duckworth
references.title: NeRF in the Wild: Neural Radiance Fields for Unconstrained Photo Collections
references.authors: Ishit Mehta Michaël Gharbi Connelly Barnes Eli Shechtman Ravi Ramamoorthi Manmohan Chandraker
references.title: Modulated periodic activations for generalizable local functional representations, 2021
references.authors: Lars Mescheder Michael Oechsle Michael Niemeyer Sebastian Nowozin Andreas Geiger
references.title: Occupancy networks: Learning 3d reconstruction in function space
references.venue: In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR),
references.authors: Mateusz Michalkiewicz Jhony Kaesemodel Pontes Dominic Jack Mahsa Baktashmotlagh Anders Eriksson
references.title: Implicit surface representations as layers in neural networks
references.venue: IEEE/CVF International Conference on Computer Vision (ICCV),
references.authors: Ben Mildenhall Pratul P. Srinivasan Matthew Tancik Jonathan T. Barron Ravi Ramamoorthi Ren Ng
references.title: Nerf: Representing scenes as neural radiance fields for view synthesis
references.venue: In ECCV,
references.authors: Jeong Joon Park Peter Florence Julian Straub Richard Newcombe Steven Lovegrove
references.title: Deepsdf: Learning continuous signed distance functions for shape representation
references.venue: In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR),
references.authors: Keunhong Park Utkarsh Sinha Jonathan T. Barron Sofien Bouaziz Dan B Goldman Steven M. Seitz Ricardo Martin-Brualla
references.title: Deformable neural radiance fields
references.venue: arXiv preprint arXiv:2011.12948,
references.authors: Songyou Peng Michael Niemeyer Lars Mescheder Marc Pollefeys Andreas Geiger
references.title: Convolutional occupancy networks
references.venue: In European Conference on Computer Vision (ECCV),
references.authors: Nasim Rahaman Aristide Baratin Devansh Arpit Felix Draxler Min Lin Fred Hamprecht Yoshua Bengio Aaron Courville
references.title: On the spectral bias of neural networks
references.venue: In International Conference on Machine Learning,
references.authors: Rajesh Rao Dana Ballard
references.title: Predictive coding in the visual cortex: a functional interpretation of some extra-classical receptive-field effects
references.venue: Nature neuroscience, 2:79–87,
references.authors: Basri Ronen David Jacobs Yoni Kasten Shira Kritchman
references.title: The convergence rate of neural networks for learned functions of different frequencies
references.venue: In NeurIPS,
references.authors: Shunsuke Saito Zeng Huang Ryota Natsume Shigeo Morishima Angjoo Kanazawa Hao Li
references.title: Pifu: Pixel-aligned implicit function for high-resolution clothed human digitization
references.venue: In The IEEE International Conference on Computer Vision (ICCV),
references.authors: Patrick Schmidt Janis Born Marcel Campen Leif Kobbelt
references.title: Distortion-minimizing injective maps between surfaces
references.venue: ACM Trans. Graph.,
references.authors: Patrick Schmidt Marcel Campen Janis Born Leif Kobbelt
references.title: Inter-surface maps via constantcurvature metrics
references.venue: ACM Trans. Graph.,
references.authors: Vincent Sitzmann Michael Zollhöfer Gordon Wetzstein
references.title: Scene representation networks: Continuous 3d-structure-aware neural scene representations
references.venue: In Advances in Neural Information Processing Systems,
references.authors: Vincent Sitzmann Julien Martel Alexander Bergman David Lindell Gordon Wetzstein
references.title: Implicit neural representations with periodic activation functions
references.venue: Advances in Neural Information Processing Systems,
references.authors: Towaki Takikawa Joey Litalien Kangxue Yin Karsten Kreis Charles Loop Derek Nowrouzezahrai Alec Jacobson Morgan McGuire Sanja Fidler
references.title: Neural geometric level of detail: Real-time rendering with implicit 3D shapes
references.venue: In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR),
references.authors: Matthew Tancik Pratul P. Srinivasan Ben Mildenhall Sara Fridovich-Keil Nithin Raghavan Utkarsh Singhal Ravi Ramamoorthi Jonathan T. Barron Ren Ng
references.title: Fourier features let networks learn high frequency functions in low dimensional domains
references.venue: NeurIPS,
references.authors: Matthew Tancik Ben Mildenhall Terrance Wang Divi Schmidt Pratul P. Srinivasan Jonathan T. Barron Ren Ng
references.title: Learned initializations for optimizing coordinate-based neural representations
references.authors: Ashish Vaswani Noam Shazeer Niki Parmar Jakob Uszkoreit Llion Jones Aidan N Gomez Ł ukasz Kaiser Illia Polosukhin
references.title: Attention is all you need
references.venue: Advances in Neural Information Processing Systems,
references.authors: Da Xu Chuanwei Ruan Evren Korpeoglu Sushant Kumar Kannan Achan
references.title: Self-attention with functional time representation learning
references.venue: In Advances in Neural Information Processing Systems,
references.authors: Zhi-Qin John Xu Yaoyu Zhang Yanyang Xiao
references.title: Training behavior of deep neural network in frequency domain
references.venue: In ICONIP,
references.authors: Alex Yu Vickie Ye Matthew Tancik Angjoo Kanazawa
references.title: pixelNeRF: Neural radiance fields from one or few images
references.venue: In CVPR,
references.authors: Qingnan Zhou Alec Jacobson
references.title: Thingi10k: A dataset of 10,000 3d-printing models
references.venue: arXiv preprint arXiv:1605.04797,
sections.heading: 1 Introduction
sections.text: Neural implicit functions have recently emerged as a powerful representation paradigm for modeling complex signals. Their continuous nature sets them apart from typical discrete representations (i.e. pixels, voxels, meshes), allowing to capture high resolution details in various domains such as images [41, 43], 3d shapes [31, 6] and radiance fields [30, 32], while retaining a reasonably compact representation. In this formulation, a deep neural network is trained with the goal of faithfully mapping input coordinates to a corresponding target domain, effectively learning the representation of signal properties such as magnitude, color, or shape occupancy.
  Implementing implicit neural representations with common neural structures, e.g., multilayer pereceptrons with ReLU activations (ReLU MLPs), proves to be challenging in the presence of signals with high frequencies. Consequently, recent works have demonstrated that deep implicit networks benefit from mapping the input coordinates [30, 43], or the intermediate features [41] to positional encodings. That is, before feeding them into a neural layer, they are first transformed to an overparameterized, high dimensional space, typically by multiple periodic functions.
  Positional encodings1 have been shown to enable highly detailed mappings of signals by MLP networks. For example, Fourier Feature Networks [43] suggested to map input coordinates of signals to a high dimensional space using sinusoidal functions. In their work, they show that the frequency
  1In this paper, we use the term “positional encodings” in lower case letters to denote the family of encoding methods that map coordinates to a higher dimensional space. Not to be confused with the term “Positional Encoding” coined by [30, 43], which refers to a particular mapping scheme in this family.
  35th Conference on Neural Information Processing Systems (NeurIPS 2021).
  of these sinusoidal encodings is the dominant factor in obtaining high quality signal reconstructions. In particular, they present compelling arguments for randomly sampling frequency values from an isotropic Gaussian distribution with a carefully selected scale, providing a striking improvement over mapping coordinates directly via standard MLPs.
  Despite the success of positional encodings, there are still some concerns left unaddressed: (i) Choosing the right frequency scale requires manual tuning, oftentimes involving a tedious parameter sweep; (ii) The frequency distribution scale may change between different inputs, and accordingly it becomes harder to tune a “one-fits-all” model for signals that are composed of a large range of frequencies (Fig. 1); (iii) Frequencies are selected for the entire input in a global, spatially invariant manner, thus missing an opportunity to better adapt to local high frequencies (Fig. 2).
  Our work investigates mitigations to the aforementioned challenges. We study the setting of positional encodings as input to implicit neural networks and present Spatially-Adaptive Progressive Encoding (SAPE). SAPE is a policy for learning implicit functions, relying on two core ideas: (i) guiding the neural optimization process by gradually unmasking signal components with increasing frequencies over time, and (ii) allowing the mask progression to propagate at different rates among local spatial portions of the signal space. To govern this process, we introduce a feedback loop to control the progression of revealed encoding frequencies as a bi-variate function of time and space.
  Our work enables MLP networks to adaptively fit a varying spectrum of fine details that previous methods struggle to capture in a single shot, without involved tuning of parameters or domain specific preprocessing. SAPE excels in learning implicit functions with a large Lipschitz constant, without sacrificing quality of details or optimization stability, in problems that require meticulous configuration to achieve convergence. To highlight the latter, in Section 5.1 we present the tasks of 2D silhouettes deformation and 3D mesh transfer – both require stable optimization from the get-go in order to avoid convergence to sub-optimal local minima.
  SAPE is encoding-agnostic: it is not limited to a specific positional encoding type. It can be easily applied to the learning process of coordinate-based neural implicit functions of various domains including images, 2D shapes, 3D occupancy maps and surfaces, showing improvement in all of them.
sections.heading: 2 Related work
sections.text: Implicit neural representations. Recent works show the capability of deep neural networks in representing different functions (e.g., 2D/3D scenes or objects) as an implicit, memory efficient continuous function [8]. These networks are used as a signed distance function (SDF) [1, 15, 31] or as an occupancy network, either binary [6, 28, 33] or soft volumetric density [26, 30, 49].
  Several works train such networks to represent a collection of 3D shapes via 3D data supervision [6, 16, 29, 28, 31], reconstruct 3D shapes [5, 11, 13, 14, 18, 42] or infer them from 2D images [22, 37, 40]. Recent works employ spatial data structures to scale the represented shapes size [25, 42].
  Positional encodings (PE). have been suggested as higher dimensional network input for various purposes. Radial basis function (RBF) networks [7] use weighted sums of embedded RBF encodings due to their symmetric property. [47] used random Fourier Features to map time spans. In natural language processing, Transformers [19, 23, 24, 46] leverage sinusoidal positional encoding to maintain the order of token sequences. Our work differs by focusing on how to encode the inputs to MLPs in order to improve network implicit representations.
  The closest works to ours are SIREN [41] and Fourier feature networks (FFN) [43]. SIREN suggests to replace the ReLU activations in the network by periodic ones. FFN [43] encodes the inputs to the network by projecting them to a higher dimensional space using a family of encoding functions. In the appendix we provide an illustration of this approach with some examplary used encodings. The analysis of [43] demonstrates that FFN improves the learning of implicit neural representation compared to other alternatives. A follow-up work in [44] accelerates the training convergence by using a meta-learned initialization of the network weights. The contemporary work of [27] extends SIREN with a modulation network, to allow representing multiple shapes within a single network.
  Concurrently to our work, Park et al. [32] extend NERF to non-rigidly deforming scenes, and BARF [21] extends NERF for cases of imperfect camera poses. Both these works show the advantage of employing coarse-to-fine annealing linearly over the frequency bandwidth of the positional encoding. The coarse-to-fine approach bears similarity to our progressive frequency approach. Different from us, these works do not use a feedback loop or spatial encoding, which, as we show in the following, further closes the gap between the regressed function and the ground truth.
  As a final remark, SAPE’s concept of feedback loop resonates with the idea of "predictive coding" in computational neuroscience [35], suggesting another angle to appreciate the rationale of our method.
sections.heading: 3 Preliminaries
sections.text: The setting of implicit neural representation is commonly formulated as learning a mapping function fθ, using a neural network, from an input of low dimensional position coordinates p ∈ Rd to an output value in the target domain. The network training data are coordinates samples as the input and an expected value of the function, or signal, as the output. Examples of such mappings include pixel locations to intensity values for images, or 3D space coordinates to binary surface occupancy labels.
  Achieving such mappings using conventional neural networks with ReLU is hard. It has been shown that ReLU networks exhibit a behaviour of spectral bias [34, 3, 36, 2, 48, 4]. Networks tend to learn low frequency functions first, as they are characterized by a global behaviour, and are therefore more stable to optimize. Spectral bias, however, also prevents networks from properly learning to fit functions that change at a high rate, e.g., functions with a large Lipschitz constant. When visual domains are concerned, this is mostly evident in delicate details missing from the network output.
  To mitigate this deficiency, recent works proposed to replace the ReLU activation layers with periodic sine functions [41] or map the input to some higher dimensional space in order to learn a mapping with high frequencies [30, 43]. In the latter approach, the input p is encoded to a high dimensional embedding by a family of functionals ei : Rd → R, such that:
  E(p) = (e1(p), e2(p), . . . , en(p)) , (1)
  where usually n d. Tancik et al. [43], for example, suggest the following encoding: EFFN (p) = ( cos(2πb>1 p), sin(2πb > 1 p), ..., cos(2πb > np), sin(2πb > np) ) , (2)
  where bi are frequency vectors randomly sampled i.i.d. from a Gaussian distribution with standard deviation σ. Other examples of positional encodings are included in the appendix.
  These embedding schemes facilitate learning of complex functions, at the price of introducing a second associated phenomenon: the spectral bias characteristic of the network is reduced. The implication in this case is twofold. Positional encodings can cause neural optimizations processes to become unstable, and consequently converge to a bad local minimum (Fig. 9). In addition, when fitting functions of varying levels of details/frequencies, neurons predicting smooth, low frequency areas are still exposed to encoding dimensions of high frequency. That, in turn, may complicate the learning process, as networks have to learn where to ignore such embedding dimensions.
  Indeed, Tancik et al. [43] advocate a careful choice of the standard deviation of frequency distribution, σ in their method, showing that using low values yields missing details in the network output, and using values that are very high results in noisy artifacts. Thereupon, for such static encodings the value of σ requires a parameter sweep per sample.
sections.heading: 4 Spatially-Adaptive Progressive Encoding (SAPE)
sections.text: Our proposed approach, SAPE, is a policy for guiding the optimization of implicit neural representations based on input coordinates. SAPE relies on the delicate balance of the two phenomena that govern the learning process: it reconciles the effects of the spectral bias and the expressiveness of high frequency encodings in a manner that benefits from both. It maintains a stable optimization, without sacrificing the ability to fit fine signal details. SAPE is composed of two key components: progressive encoding and spatial adaptivity, which allow it to be less sensitive to the encoding frequency bandwidth, i.e., the choice of the standard deviation in the case of [43] (Fig. 4). Next, we detail each of the mechanisms that compose SAPE. The full algorithm is included in the appendix.
  Without loss of generality, in our description below we assume the encoding functionals are sorted by their respective Lipschitz constants (LC), i.e. LC(e1) ≤ LC(e2) ≤ · · · ≤ LC(en). For example, we sort the Fourier features encoding by the frequency value ‖bi‖. We set the first d dimensions to the identity encodings: ei(p) = pi, ∀i ≤ d, thereby exposing the network to the original input position coordinates p ∈ Rd as well. Progressive encoding. A core part of SAPE is progressively fitting the frequencies. The first layer of the network encodes an input position p with a set of functionals ei. In order to use only part of the functionals at different stages of training, it multiplies the encoding dimensions with a soft mask, which progresses as a function of the optimization iteration t:
  Eprog(p, t) = (ê0(p, t), ê1(p, t), ê2(p, t), . . . , ên(p, t)) = α(t) >E(p), (3)
  where êi(p, t) = αi(t) ei(p) is the progression control for the encoding functional ei, regulated by the elements of masking vector α(t) ∈ [0, 1]n. During the optimization, we progressively reveal encoding features in a manner that allows the encoding neurons to be tuned to different states according to the value of αi(t), where αi(t) = 1 corresponds to the state on, αi(t) = 0 means off and 0 < αi(t) < 1 denotes partially on (Fig. 3).
  We define a policy Φ that controls the progression of the mask vector α(t) by α(t + 1) = Φ [α(t)|α(0)] . In this work, we use the following progression rule for the mask of the ith encoding dimension at time step t:
  ΦP [α(t)] : αi(t+ 1) = clamp ( t− τ · (i− d) τ , 0, 1 ) , if i > d
  1, otherwise, (4)
  where τ = T/2n represents the number of iterations each encoding dimension takes to progress from 0 to 1 and T is the maximal number of iterations in the optimization. Essentially, ΦP performs a linear progression sequentially for each encoding i until it is fully exposed, s.t. αi(t) = 1. When a certain encoding dimension mask achieves saturation, we continue to the following one. We allow a technical exception to this rule, for proper progression of correlated masks of encoding dimensions sharing the same Lipschitz constant, e.g., pairs of cos, sin in the encoding in Eq. (2). For initialization, the first d positional encoding masks are fully exposed right from the beginning, and in our setting, correspond with the identity encoding functionals.
  Notice that previous methods, which introduce all encoding dimensions to the network at once, can now be formulated as a specific case in our framework, where α(t+ 1) = 1.
  Spatial adaptivity. When t > T/2, the progression policy ΦP achieves global saturation in terms of revealing encoding dimensions, so that the masking vector becomes α(t) = 1. For signals of global-like characteristics, e.g., have a low LC, the network is compelled to learn how to cope with undesirable encoding dimensions of high frequency to regress a smooth, slowly changing signal. This can lead to sub-optimal outputs, often visible in the form of visual artifacts (Fig. 1, mid-bottom).
  Given loss function L and convergence threshold ε, a simple improvement to policy ΦP may apply early stopping when L < ε. However, for signals with varying levels of detail as a function of spatial position p, this improvement is not sufficient: early stopping does not occur due to areas characterized by high frequencies. Relying on a global threshold ε is inherently a sub-optimal decision, as when learning an implicit neural function with positional encoding, different spatial segments of the signal in question may converge at different rates (Fig. 2).
  To solve this problem, we set the mask vector to be spatially sensitive: α(t,p). By design, the new policy ΦSA progresses the encoding of each spatial location p separately, per encoding dimension i. To simplify our explanation, in the current setting we assume the signal can be discretized to distinct spatial segments on a regular grid, each tracked independently (e.g., pixels, voxels). The progressive, spatially adaptive policy is controlled by a feedback loop, where the independent regression loss of each signal segment is used to control the progression:
  ΦSA [α(t,p)] : α(t+ 1,p) = { ΦP (α(t)) , if L(t,p) ≥ ε α(t,p), otherwise
  (5)
  For implicit functions, at inference time we assume signals can be regressed with “pseudo-continuous” coordinates p. Therefore, to support coordinates p that did not appear during training and have no mask recordings, we extend the estimated parameters of α(T,p) continuously over the entire input domain by a linear interpolation.
  One common example that benefits from spatial adaptivity is natural 2D image containing blurry, out-of-focus areas in the background together with sharp, detailed foreground objects. To shed more light on the behaviour of our algorithm, Figs. 6 and 8 demonstrate examples of spatial heat maps, highlighting the maximal encoding frequency achieved per spatial location, upon convergence.
  a. Forward mask
  b. Loss feedback
  Sparse grid sampling. We now extend our method to cases where model fθ is optimized by coordinates p that do not lie on a regular grid, or when it is simply infeasible to store parameters α(t,p) per sampled coordinate p during training. In this setting, we discretize the input domain and store parameters αu in a sparse grid G, where for a grid of resolution rd ⊆ Rd, we denote u as the multidimensional grid coordinates. In the forward pass, we obtain the original encoding mask α(t,p) per sampled point p by encoding a linear interpolation of the masking parameters over the nearest grid nodes (see the inset (a) on the left):
  α(t,p) = ∑
  u∈NG(p)
  wp,uα(t,u), (6)
  where NG(p) are the neighboring grid coordinates in the vicinity of p, and wp,u are the interpolation weights for a sample p over the multidimensional grid G such that ∑ u∈NG wp,u = 1. During training, the loss L(t,u) at time t for a given grid point u is accumulated over all sampled points p affected by u (inset (b) above):
  L(t,u) = 1∑ p wp,u ∑ p wp,uL(t,p). (7)
  Here, L(t,p) is the loss for the original coordinate p. To obtain the overall training loss, we sum over all the used points for training at time t. We note that the linear interpolation used during the forward pass has an added benefit: since only local neighbors participate per update of the original coordinate p, sparse weights allow for efficient accumulations during forward and backward pass updates.
sections.heading: 5 Experiments
sections.text: We evaluate SAPE on a variety of common 2D and 3D regressions tasks. In addition, we demonstrate how it may be used to improve the tasks of deforming 2D silhouettes and transferring mesh connectivity between 3D shapes. We demonstrate our results using the Fourier feature encoding [43] (Eq. (2)). We emphasize that SAPE is agnostic to the encoding used and applicable to other mapping schemes as well. More examples, as well as the full implementation details appear in the appendix.
sections.heading: 5.1 Evaluations
sections.text: We test SAPE on a variety of problems: regression tasks optimized by a direct supervision and geometric tasks optimized by an indirect supervision. All configurations employ 256 unique frequency encodings sampled from a Gaussian distribution. For fair evaluation, the distribution scale and number of neural layers remain the same for all inputs in the same problem setting. However, hyperparameter values may vary per problem. For convergence threshold ε, we set the values of 1e−3 for regression tasks and 1e−2 for geometric tasks. See the appendixfor full description of implementation details. We compare the settings of 6 MLP configurations:
  1) No encoding: Basic ReLU MLP without encoding. 2) SIREN: An MLP with sine activations based on the implementation of Sitzmann et al. [41]. 3) RBF-grid: An MLP with repeated radial basis function as a first encoding layer. 4) SAPE + RBF-grid. 5) FFN: An MLP network with Fourier features as the first encoding layer, see Eq. (2). 6) SAPE + FFN.
  The bandwidths of encoding functions in 3) and 5) are optimally selected by a grid search over a validation set or taken from a public implementation, depending on the task. For the SAPE variants we use double the σ of these bandwidths, as SAPE is not sensitive to a particular value, as long as it allows high frequency encodings. The quantitative results are summarized in Tables 1 and 2. Below is an overview of each task.
  2D image regression. In this task we optimize the networks to map 2D input pixel coordinates, normalized to [−1, 1]2, to their RGB values. We conduct the evaluation on the same test sets as Tancik et al. [43], which contain a dataset of natural images and a synthetic dataset of text images. Similar to them, the network is trained on regularly spaced grid containing 25% of the pixels. We use the evaluation metric of PSNR over the entire image, compared to the ground truth image. Quantitative results are reported in Table 1 on the left.
  3D occupancy. In this task, we use a similar setting to occupancy networks [28]: the model is trained to classify an input 3D coordinate for being inside or outside the training shape. For training, we sample 9 million points divided into 3 equal groups: uniformly sampled points in [−1, 1]3, surface points perturbed with random Gaussian noise vectors using σ = 0.1 and σ = 0.01. We evaluate the quality of the result by estimating the intersection-over-union (IoU) with respect to the ground truth shape by sampling additional 1e6 random points and 1e6 more challenging samples near the surface, and report the average IoU score.
  We compare the networks on two test sets. The first set is composed of 10 selected models from the Thingi10K dataset [50]. The second, more challenging test set is composed of 4 models from TurboSquid2. The quantitative results are summarized in Table 1, middle section. Qualitative results are shown in the appendix.
  2D silhouettes. Here, we optimize an MLP to deform a unit circle represented as a polyline to a target 2D point cloud of a silhouette shape. Fig. 9 shows a number of examples of such target shapes. We start by calibrating the MLP to learn a simple mapping from p ∈ [0, 1] to the 2D unit circle: f(p) = (cos(2πp), sin(2πp)). We then optimize the mapping function by minimizing the symmetric chamfer distance between the network output and the silhouette.
  To evaluate the performance of the different methods, we test the networks on 20 shapes from the MPEF7 dataset [20] and measure the intersection over union between the resulting shape and the target shape; the results are reported in Table 1. Fig. 9 shows qualitative results of this task and snapshots from the optimization process. Due to the expressiveness of both FFN and SIREN networks, the chamfer loss causes distortions in the early stage of the optimization that cannot be recovered, leading to undesired results. The coarse-to-fine optimization of SAPE allows both avoiding distortions and matching high frequencies of the target shape.
  2https://www.turbosquid.com
  3D mesh transfer. In this task, we would like to transfer a 3D source mesh to a target shape [45, 9, 12, 38, 39], which is represented by a mesh or a point cloud. The MLP receives the vertices of the source mesh and outputs transformed vertices, such that together with the source tessellation, the optimized mesh fits the target shape while respecting the structure of the source mesh in terms of distortion. In addition, we may utilize a set of marked correspondence points between the input shapes that enable the estimation of an initial affine transformation between the source and the target, followed by a biharmonic deformation [17]. The optimization loss for this task is composed of two terms: A distance loss that measures the symmetric chamfer distance between the optimized mesh and the target shape, and a structural term that measures the discrete conformal energy between the optimized mesh and the source mesh.
  We test the different networks on 6 pairs of meshes and evaluate the results by measuring the chamfer and Hausdorff distances between the target shape and the output mesh. The distortion of the transferred meshing is measured by Dirichlet energy with respect to the source mesh. Table 2 shows the quantitative results. Qualitative results are shown in Fig. 5.
  Standard MLPs struggle to fit the target shape and remain close to the source mesh, therefore obtaining low Dirichlet energy but also high Chamfer and Hausdorff distances and thus failing the task. SAPE’s feedback specifies the regions on the mesh that are distanced from the target shape. That allows the optimization to gradually increase the frequencies used in high curvature areas while avoiding large distortions at the beginning of the optimization, when global deformations take place. For that reason, contrary to other methods such as FFN and SIREN, SAPE is able to avoid solutions of bad local minima and reconstruct the target signal better.
sections.heading: 5.2 Ablation
sections.text: Progressive vs. non-progressive. Similar to the setting described in Section 5.1, we train an MLP on randomly sampled pixels of images. We map 2D pixel coordinates to RGB values and measure the PSNR over all pixels compared to the ground truth image.
  Fig. 6 and Fig. 7 (left) show a comparison between the performance of FFN with and without SAPE when training on 25% of the pixels. Our results show that due to progressive-spatial adjustment of the encoding, SAPE is less sensitive to tuning of the standard deviation (σ) of distribution of frequencies bi. By contrast, FFN suffers from a fragile trade-off between underfitting and overfitting, which results in either blurry pixels in high frequency regions or noise in the smooth background areas.
  A similar phenomena is shown in the 3D example in Fig. 10. In this task, we train an occupancy network using the setting described in Section 5.1. For regulating the level of encoding across the scene our method uses a voxel map of resolution 1283 for spatial encoding. This map is updated based on feedback loss during training. Without SAPE, low frequency encodings cannot represent detailed structures like the car object in the scene. Increasing the frequency bandwidth results in noisy surfaces and undesired blob artifacts in empty spaces. By contrast, SAPE achieves better representation of detailed regions, as well as of smooth surfaces.
  Spatial progressive vs. non-spatial progressive. To demonstrate the advantage of using spatially adaptive encoding, we compare it to a variant of SAPE that globally progresses the frequency level of encodings for all spatial areas equally, and converges when all training samples fit. Fig. 2 shows the advantage of using the spatial encoding in SAPE for a 1D regression task. In addition, Table 1
  shows quantitative improvement gained by SAPE applied on FF, compared to a global progression (Progressive FF).
  Note that the non-spatial variant is equivalent to SAPE with a spatial grid of resolution one. Fig. 8 and Fig. 7 (right) show the benefit of the spatial encoding component for a 2D image regression task (similar to the one above). We also show how increasing the grid resolution affects the quality of the result in a 2D image regression task. We observe that the quality stays at its highest when the spatial grid resolution approximates the sampling ratio of coordinates during training.
sections.heading: 6 Limitations and Future Work
sections.text: When SAPE is employed with a sampling grid, the chosen resolution provides a trade-off between memory and quality. Choosing a grid of low resolution may yield sub-optimal outputs (see examples in the appendix). As SAPE’s varying frequencies sampling grid bears some resemblance to multiresolution analysis and wavelet transforms, future extensions to SAPE may solve this issue by maintaining the masking parameters in a sparse structure such as an Octree [42] or GMM [10].
  When using very high frequencies, SAPE may enhance inherent output noise, as the maximal frequency encoding gets picked for the noisy portions of the signal.
  In this work we investigated a linear progression policy for unmasking frequency encodings. Followup works may extend our scheme to a more optimized curriculum, for example, using meta-learning.
sections.heading: 7 Conclusions
sections.text: We presented a policy for improving the quality of signals learned by coordinate based optimizations. We rely on two major contributions: progressive introduction of positional encodings, and spatial adaptivity, which allows different rates of progression per signal location. Our method is simple to implement and improves the implicit neural representation result of MLP networks in various tasks.
  Given the surge of applications that use positional encodings, and the ease of deploying SAPE, we believe our approach will be useful for many problems in the vision and graphics domains. Specifically, the fact that SAPE is encoding-agnostic and insensitive to most encoding hyperparameters provides stable training and facilitates the use of positional encodings in novel tasks.
sections.heading: 8 Broader Impact Statement
sections.text: SAPE is a method for boosting the accuracy of learned implicit neural functions. It can be leveraged to enable new applications or improve various downstream tasks that involve precise processing of signals, like segmentation, or reconstruction of complex signals, such as streaming video or signed distanced functions. Beyond the domains of vision and graphics, we conjecture SAPE might also benefit applications in speech, audio, and signal processing in general. Depending on their use, all mentioned applications may have a positive or negative impact. SAPE can be used to facilitate novel applications that require accurate recovery of signals, like 3D reconstruction for AR and VR. It can also be misused for the purpose of generating fake media, including audio, video and images.
sections.heading: Acknowledgements
sections.text: We would like to thank Hao (Richard) Zhang, Hadar Averbuch-Elor and Rinon Gal for their insightful comments, and the anonymous reviewers for their helpful remarks. This work was supported in part by the Israel Science Foundation grant no. 2492/20 and by the ERC-StG grant no. 757497 (SPADE).
title: SAPE: Spatially-Adaptive Progressive Encoding for Neural Optimization