Towards flexible perception with visual memory
Training a neural network is a monolithic endeavor, akin to carving knowledge
into stone: once the process is completed, editing the knowledge in a network
is nearly impossible, since all information is distributed across the network's
weights. We here explore a simple, compelling alternative by marrying the
representational power of deep neural networks with the flexibility of a
database. Decomposing the task of image classification into image similarity
(from a pre-trained embedding) and search (via fast nearest neighbor retrieval
from a knowledge database), we build a simple and flexible visual memory that
has the following key capabilities: (1.) The ability to flexibly add data
across scales: from individual samples all the way to entire classes and
billion-scale data; (2.) The ability to remove data through unlearning and
memory pruning; (3.) An interpretable decision-mechanism on which we can
intervene to control its behavior. Taken together, these capabilities
comprehensively demonstrate the benefits of an explicit visual memory. We hope
that it might contribute to a conversation on how knowledge should be
represented in deep vision models -- beyond carving it in ``stone'' weights.
66becda6a474a917fc5cf8bf
Robert Geirhos
claimed_verified
2025-01-17T17:22:40.038Z
673bbe0d7dfcdedd52619ec2
/avatars/531a44f05d0c738bbe3e028c76c2e948.svg
Robert Geirhos
user
rgeirhos
66becda6a474a917fc5cf8c0
Priyank Jaini
66becda6a474a917fc5cf8c1
Austin Stone
66becda6a474a917fc5cf8c2
Sourabh Medapati
claimed_verified
2024-08-16T21:02:56.175Z
648888b165e25cc87b6ab206
/avatars/c972e4ff17a446c4d2f803fe9315ce12.svg
Sourabh Medapati
user
smedapati
66becda6a474a917fc5cf8c3
Xi Yi
66becda6a474a917fc5cf8c4
George Toderici
66becda6a474a917fc5cf8c5
Abhijit Ogale
66becda6a474a917fc5cf8c6
Jonathon Shlens
Towards flexible perception with visual memory
Training a neural network is a monolithic endeavor, akin to carving knowledge
into stone: once the process is completed, editing the knowledge in a network
is nearly impossible, since all information is distributed across the network's
weights. We here explore a simple, compelling alternative by marrying the
representational power of deep neural networks with the flexibility of a
database. Decomposing the task of image classification into image similarity
(from a pre-trained embedding) and search (via fast nearest neighbor retrieval
from a knowledge database), we build a simple and flexible visual memory that
has the following key capabilities: (1.) The ability to flexibly add data
across scales: from individual samples all the way to entire classes and
billion-scale data; (2.) The ability to remove data through unlearning and
memory pruning; (3.) An interpretable decision-mechanism on which we can
intervene to control its behavior. Taken together, these capabilities
comprehensively demonstrate the benefits of an explicit visual memory. We hope
that it might contribute to a conversation on how knowledge should be
represented in deep vision models -- beyond carving it in ``stone'' weights.
A visual memory system that combines deep neural networks with databases to offer flexible data addition, removal, and interpretable decision-making, enhancing knowledge representation in visual models.
deep neural networks pre-trained embedding nearest neighbor retrieval knowledge database visual memory memory pruning unlearning interpretable decision-mechanism