Towards flexible perception with visual memory
<|reference_start|>Towards flexible perception with visual memory: Training a neural network is a monolithic endeavor, akin to carving knowledge into stone: once the process is completed, editing the knowledge in a network is nearly impossible, since all information is distributed across the network's weights. We here explore a simple, compelling alternative by marrying the representational power of deep neural networks with the flexibility of a database. Decomposing the task of image classification into image similarity (from a pre-trained embedding) and search (via fast nearest neighbor retrieval from a knowledge database), we build a simple and flexible visual memory that has the following key capabilities: (1.) The ability to flexibly add data across scales: from individual samples all the way to entire classes and billion-scale data; (2.) The ability to remove data through unlearning and memory pruning; (3.) An interpretable decision-mechanism on which we can intervene to control its behavior. Taken together, these capabilities comprehensively demonstrate the benefits of an explicit visual memory. We hope that it might contribute to a conversation on how knowledge should be represented in deep vision models -- beyond carving it in "stone" weights.<|reference_end|>
@article{geirhos2024towards,
    title={Towards flexible perception with visual memory},
    author={Robert Geirhos, Priyank Jaini, Austin Stone, Sourabh Medapati, Xi Yi,
  George Toderici, Abhijit Ogale, Jonathon Shlens},
    journal={arXiv preprint arXiv:2408.08172},
    year={2024},
    archivePrefix={arXiv},
    eprint={2408.08172},
    primaryClass={cs.CV cs.AI cs.LG}
}