title: Scaling Face Interaction Graph Networks to Real World Scenes
summary: Accurately simulating real world object dynamics is essential for various
  applications such as robotics, engineering, graphics, and design. To better
  capture complex real dynamics such as contact and friction, learned simulators
  based on graph networks have recently shown great promise. However, applying
  these learned simulators to real scenes comes with two major challenges: first,
  scaling learned simulators to handle the complexity of real world scenes which
  can involve hundreds of objects each with complicated 3D shapes, and second,
  handling inputs from perception rather than 3D state information. Here we
  introduce a method which substantially reduces the memory required to run
  graph-based learned simulators. Based on this memory-efficient simulation
  model, we then present a perceptual interface in the form of editable NeRFs
  which can convert real-world scenes into a structured representation that can
  be processed by graph network simulator. We show that our method uses
  substantially less memory than previous graph-based simulators while retaining
  their accuracy, and that the simulators learned in synthetic environments can
  be applied to real world scenes captured from multiple camera angles. This
  paves the way for expanding the application of learned simulators to settings
  where only perceptual information is available at inference time.
paper_authors._id: 65af53dfb64e1c2389110561
paper_authors.name: Tatiana Lopez-Guevara
paper_authors._id: 65af53dfb64e1c2389110562
paper_authors.name: Yulia Rubanova
paper_authors._id: 65af53dfb64e1c2389110563
paper_authors.name: William F. Whitney
paper_authors._id: 65af53dfb64e1c2389110564
paper_authors.name: Tobias Pfaff
paper_authors._id: 65af53dfb64e1c2389110565
paper_authors.name: Kimberly Stachenfeld
paper_authors._id: 65af53dfb64e1c2389110566
paper_authors.name: Kelsey R. Allen
paper_title: Scaling Face Interaction Graph Networks to Real World Scenes
paper_summary: Accurately simulating real world object dynamics is essential for various
  applications such as robotics, engineering, graphics, and design. To better
  capture complex real dynamics such as contact and friction, learned simulators
  based on graph networks have recently shown great promise. However, applying
  these learned simulators to real scenes comes with two major challenges: first,
  scaling learned simulators to handle the complexity of real world scenes which
  can involve hundreds of objects each with complicated 3D shapes, and second,
  handling inputs from perception rather than 3D state information. Here we
  introduce a method which substantially reduces the memory required to run
  graph-based learned simulators. Based on this memory-efficient simulation
  model, we then present a perceptual interface in the form of editable NeRFs
  which can convert real-world scenes into a structured representation that can
  be processed by graph network simulator. We show that our method uses
  substantially less memory than previous graph-based simulators while retaining
  their accuracy, and that the simulators learned in synthetic environments can
  be applied to real world scenes captured from multiple camera angles. This
  paves the way for expanding the application of learned simulators to settings
  where only perceptual information is available at inference time.
paper_ai_summary: A method for reducing memory usage in graph-based learned simulators and a perceptual interface using editable NeRFs enable the application of these simulators to real-world scenes from perceptual data.
paper_ai_keywords: graph networks learned simulators memory-efficient NeRFs perceptual interface real-world scenes perceptual information