title: Towards flexible perception with visual memory
summary: Training a neural network is a monolithic endeavor, akin to carving knowledge
  into stone: once the process is completed, editing the knowledge in a network
  is nearly impossible, since all information is distributed across the network's
  weights. We here explore a simple, compelling alternative by marrying the
  representational power of deep neural networks with the flexibility of a
  database. Decomposing the task of image classification into image similarity
  (from a pre-trained embedding) and search (via fast nearest neighbor retrieval
  from a knowledge database), we build a simple and flexible visual memory that
  has the following key capabilities: (1.) The ability to flexibly add data
  across scales: from individual samples all the way to entire classes and
  billion-scale data; (2.) The ability to remove data through unlearning and
  memory pruning; (3.) An interpretable decision-mechanism on which we can
  intervene to control its behavior. Taken together, these capabilities
  comprehensively demonstrate the benefits of an explicit visual memory. We hope
  that it might contribute to a conversation on how knowledge should be
  represented in deep vision models -- beyond carving it in ``stone'' weights.
paper_authors._id: 66becda6a474a917fc5cf8bf
paper_authors.name: Robert Geirhos
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-01-17T17:22:40.038Z
paper_authors._id: 673bbe0d7dfcdedd52619ec2
paper_authors.avatarUrl: /avatars/531a44f05d0c738bbe3e028c76c2e948.svg
paper_authors.fullname: Robert Geirhos
paper_authors.type: user
paper_authors.user: rgeirhos
paper_authors._id: 66becda6a474a917fc5cf8c0
paper_authors.name: Priyank Jaini
paper_authors._id: 66becda6a474a917fc5cf8c1
paper_authors.name: Austin Stone
paper_authors._id: 66becda6a474a917fc5cf8c2
paper_authors.name: Sourabh Medapati
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-08-16T21:02:56.175Z
paper_authors._id: 648888b165e25cc87b6ab206
paper_authors.avatarUrl: /avatars/c972e4ff17a446c4d2f803fe9315ce12.svg
paper_authors.fullname: Sourabh Medapati
paper_authors.type: user
paper_authors.user: smedapati
paper_authors._id: 66becda6a474a917fc5cf8c3
paper_authors.name: Xi Yi
paper_authors._id: 66becda6a474a917fc5cf8c4
paper_authors.name: George Toderici
paper_authors._id: 66becda6a474a917fc5cf8c5
paper_authors.name: Abhijit Ogale
paper_authors._id: 66becda6a474a917fc5cf8c6
paper_authors.name: Jonathon Shlens
paper_title: Towards flexible perception with visual memory
paper_summary: Training a neural network is a monolithic endeavor, akin to carving knowledge
  into stone: once the process is completed, editing the knowledge in a network
  is nearly impossible, since all information is distributed across the network's
  weights. We here explore a simple, compelling alternative by marrying the
  representational power of deep neural networks with the flexibility of a
  database. Decomposing the task of image classification into image similarity
  (from a pre-trained embedding) and search (via fast nearest neighbor retrieval
  from a knowledge database), we build a simple and flexible visual memory that
  has the following key capabilities: (1.) The ability to flexibly add data
  across scales: from individual samples all the way to entire classes and
  billion-scale data; (2.) The ability to remove data through unlearning and
  memory pruning; (3.) An interpretable decision-mechanism on which we can
  intervene to control its behavior. Taken together, these capabilities
  comprehensively demonstrate the benefits of an explicit visual memory. We hope
  that it might contribute to a conversation on how knowledge should be
  represented in deep vision models -- beyond carving it in ``stone'' weights.
paper_ai_summary: A visual memory system that combines deep neural networks with databases to offer flexible data addition, removal, and interpretable decision-making, enhancing knowledge representation in visual models.
paper_ai_keywords: deep neural networks pre-trained embedding nearest neighbor retrieval knowledge database visual memory memory pruning unlearning interpretable decision-mechanism