title: Real-Time Neural Appearance Models
summary: We present a complete system for real-time rendering of scenes with complex
  appearance previously reserved for offline use. This is achieved with a
  combination of algorithmic and system level innovations.
    Our appearance model utilizes learned hierarchical textures that are
  interpreted using neural decoders, which produce reflectance values and
  importance-sampled directions. To best utilize the modeling capacity of the
  decoders, we equip the decoders with two graphics priors. The first prior --
  transformation of directions into learned shading frames -- facilitates
  accurate reconstruction of mesoscale effects. The second prior -- a microfacet
  sampling distribution -- allows the neural decoder to perform importance
  sampling efficiently. The resulting appearance model supports anisotropic
  sampling and level-of-detail rendering, and allows baking deeply layered
  material graphs into a compact unified neural representation.
    By exposing hardware accelerated tensor operations to ray tracing shaders, we
  show that it is possible to inline and execute the neural decoders efficiently
  inside a real-time path tracer. We analyze scalability with increasing number
  of neural materials and propose to improve performance using code optimized for
  coherent and divergent execution. Our neural material shaders can be over an
  order of magnitude faster than non-neural layered materials. This opens up the
  door for using film-quality visuals in real-time applications such as games and
  live previews.
paper_authors._id: 645480ec7f7b2bed1a0391a1
paper_authors.name: Tizian Zeltner
paper_authors._id: 645480ec7f7b2bed1a0391a2
paper_authors.name: Fabrice Rousselle
paper_authors._id: 645480ec7f7b2bed1a0391a3
paper_authors.name: Andrea Weidlich
paper_authors._id: 645480ec7f7b2bed1a0391a4
paper_authors.name: Petrik Clarberg
paper_authors._id: 645480ec7f7b2bed1a0391a5
paper_authors.name: Jan Novák
paper_authors._id: 645480ec7f7b2bed1a0391a6
paper_authors.name: Benedikt Bitterli
paper_authors._id: 645480ec7f7b2bed1a0391a7
paper_authors.name: Alex Evans
paper_authors._id: 645480ec7f7b2bed1a0391a8
paper_authors.name: Tomáš Davidovič
paper_authors._id: 645480ec7f7b2bed1a0391a9
paper_authors.name: Simon Kallweit
paper_authors._id: 645480ec7f7b2bed1a0391aa
paper_authors.name: Aaron Lefohn
paper_title: Real-Time Neural Appearance Models
paper_summary: We present a complete system for real-time rendering of scenes with complex
  appearance previously reserved for offline use. This is achieved with a
  combination of algorithmic and system level innovations.
    Our appearance model utilizes learned hierarchical textures that are
  interpreted using neural decoders, which produce reflectance values and
  importance-sampled directions. To best utilize the modeling capacity of the
  decoders, we equip the decoders with two graphics priors. The first prior --
  transformation of directions into learned shading frames -- facilitates
  accurate reconstruction of mesoscale effects. The second prior -- a microfacet
  sampling distribution -- allows the neural decoder to perform importance
  sampling efficiently. The resulting appearance model supports anisotropic
  sampling and level-of-detail rendering, and allows baking deeply layered
  material graphs into a compact unified neural representation.
    By exposing hardware accelerated tensor operations to ray tracing shaders, we
  show that it is possible to inline and execute the neural decoders efficiently
  inside a real-time path tracer. We analyze scalability with increasing number
  of neural materials and propose to improve performance using code optimized for
  coherent and divergent execution. Our neural material shaders can be over an
  order of magnitude faster than non-neural layered materials. This opens up the
  door for using film-quality visuals in real-time applications such as games and
  live previews.
paper_ai_keywords: learned hierarchical textures neural decoders reflectance values importance-sampled directions graphics priors learned shading frames mesoscale effects microfacet sampling distribution anisotropic sampling level-of-detail rendering material graphs neural representation hardware accelerated tensor operations real-time path tracer neural material shaders coherent and divergent execution neural materials