We present a new approach to 3D object representation where the geometry of an object is encoded directly into the weights and biases of a second 'mapping' network. This mapping network can be used to reconstruct an object by applying its encoded transformation to points randomly sampled from a simple geometric space, such as the unit sphere. Our experiments examine the effectiveness of our method on a subset of the ShapeNet dataset. We find that this representation can reconstruct objects with accuracy equal to or exceeding state-of-the-art methods with orders of magnitude fewer parameters. Our smallest reconstruction network has only about 7000 parameters and shows reconstruction quality on par with stateof-the-art object representation architectures with millions of parameters. Further experiments show that the space of functions learned meaningfully captures the features of the encoded objects, enabling feature mixing through the composition of these functions.
This work is focused on learning 3D object representations (decoders) that can be computed more efficiently than existing methods.  The computational inefficiency of these methods is that you learn a (big) fixed decoder for all objects (all z latents), and then need to apply it individually on either each point cloud point you want to produce, or each voxel in the output (this problem exists for both the class of methods that deform a uniform distribution R^3 -> R^3 a la FoldingNet, or directly predict the 3D function R^3 -> R e.g. DeepSDF). The authors propose that the encoder directly predict the weights and biases of a decoder network that, since it is specific to the particular object being reconstructed, can be much smaller and thus much cheaper to compute.
The authors then note the fact that their method lacks a continuous latent space that allows for interpolation, as provided by existing (VAE-like) methods. They propose to solve this by learning an MLP that produces the output by recurrent application, and then composing subapplications of different networks as a type of interpolation.
-------------------
I like this work, it addresses a real problem in a number of models for 3D representation learning (similar models are also used for e.g. cryo-EM reconstruction). While the fast weights approach is not totally original, its application to this problem is novel and very well-suited to it. I was a bit surprised by just how much the decoder network could be shrunk by using fast weights. 
The paper is also quite well written. I especially like how Section 2 synthesizes existing work into model categories which make it easier to think about their relationships. I also think the explanation in Sec. 3.2, while kind of obvious, is a nice way think about decoder vs. fast weights.
I like that the authors are straightforward about the deficiency of the method (i.e. that you can't interpolate in latent space). Their proposed solution of functional composition is exceedingly clever but in my opinion too impractical to really be useful. It adds extra complexity, requires you to do function composition which may be less expressive and takes more coomputation, etc. And to what end? The purpose of generative models is not to interpolate per se; the interpolation is really a sanity check that the model is capturing the underlying distribution rather than just memorizing training examples. The function composition doesn't capture that. I think the authors should just acknowledge that you can't soundly *sample* from their generative model the way e.g. VAE or GAN allows (their function composition is not a sampling method). But I think there are lots of useful things you can do without that capability, e.g. do 3D point cloud completion, go image -> structure, etc. I think this function composition angle should be deemphasized in the title/abstract, but I think the paper stands  reasonably on its own without that.
Nits:
- In Figure 2 it's pretty hard to see the differences between the methods. What exactly is being visualized here? DeepSDF shold be visualizing surface normals vs. HOF which is point clouds, right?
- For predicting a deformation R^3 -> R^3 function composition sort of makes sense, but how generalizable is this approach e.g. to directly predicting a function R^3 -> R (a la DeepSDF)? I think there are ways this function composition approach could generalize, e.g. using skip connections and layer dropout (which encourages layers to be composable).