Squeeze-and-Remember Block
<|reference_start|>Squeeze-and-Remember Block: Convolutional Neural Networks (CNNs) are important for many machine learning tasks. They are built with different types of layers: convolutional layers that detect features, dropout layers that help to avoid over-reliance on any single neuron, and residual layers that allow the reuse of features. However, CNNs lack a dynamic feature retention mechanism similar to the human brain's memory, limiting their ability to use learned information in new contexts. To bridge this gap, we introduce the "Squeeze-and-Remember" (SR) block, a novel architectural unit that gives CNNs dynamic memory-like functionalities. The SR block selectively memorizes important features during training, and then adaptively re-applies these features during inference. This improves the network's ability to make contextually informed predictions. Empirical results on ImageNet and Cityscapes datasets demonstrate the SR block's efficacy: integration into ResNet50 improved top-1 validation accuracy on ImageNet by 0.52% over dropout2d alone, and its application in DeepLab v3 increased mean Intersection over Union in Cityscapes by 0.20%. These improvements are achieved with minimal computational overhead. This show the SR block's potential to enhance the capabilities of CNNs in image processing tasks.<|reference_end|>
@article{cakaj2024squeeze-and-remember,
    title={Squeeze-and-Remember Block},
    author={Rinor Cakaj, Jens Mehnert, Bin Yang},
    journal={arXiv preprint arXiv:2410.00823},
    year={2024},
    archivePrefix={arXiv},
    eprint={2410.00823},
    primaryClass={cs.CV cs.LG}
}