Siamese Attention Networks
Attention operators have been widely applied on data of various orders and dimensions such as texts, images, and videos. One challenge of applying attention operators is the excessive usage of computational resources. This is due to the usage of dot product and softmax operator when computing similarity scores. In this work, we propose the Siamese similarity function that uses a feed-forward network to compute similarity scores. This results in the Siamese attention operator (SAO). In particular, SAO leads to a dramatic reduction in the requirement of computational resources. Experimental results show that our SAO can save 94% memory usage and speed up the computation by a factor of 58 compared to the regular attention operator. The computational advantage of SAO is even larger on higher-order and higher-dimensional data. Results on image classification and restoration tasks demonstrate that networks with SAOs are as effective as models with regular attention operator, while significantly outperform those without attention operators.
SUMMARY: A new similarity function replacing the dot product of key and query in attention modules - instead take a shared-weighted sum Good paper, sound theory, very clear explanations. Literature review was sufficient to explain the problem and underlying theory. Results look convincing, although they cannot be verified unless code is shared. Reasonable direction of exploration - there are several possible similarity functions, this paper explores one of them that offers significantly less computational resources, which are essential for on-device applications. Thorough exploration of this idea was done and I am convinced this is a good alternative to regular attention. All the rest of the paper was to incorporate this new method in different tasks in different architectures using or not using attention, and seeing the differences in terms of computational resources. Well thought experiments and results. Again, cannot be verified unless code is shared.
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
Siamese Attention Networks
Attention operators have been widely applied on data of various orders and dimensions such as texts, images, and videos. One challenge of applying attention operators is the excessive usage of computational resources. This is due to the usage of dot product and softmax operator when computing similarity scores. In this work, we propose the Siamese similarity function that uses a feed-forward network to compute similarity scores. This results in the Siamese attention operator (SAO). In particular, SAO leads to a dramatic reduction in the requirement of computational resources. Experimental results show that our SAO can save 94% memory usage and speed up the computation by a factor of 58 compared to the regular attention operator. The computational advantage of SAO is even larger on higher-order and higher-dimensional data. Results on image classification and restoration tasks demonstrate that networks with SAOs are as effective as models with regular attention operator, while significantly outperform those without attention operators.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



SUMMARY: A new similarity function replacing the dot product of key and query in attention modules - instead take a shared-weighted sum Good paper, sound theory, very clear explanations. Literature review was sufficient to explain the problem and underlying theory. Results look convincing, although they cannot be verified unless code is shared. Reasonable direction of exploration - there are several possible similarity functions, this paper explores one of them that offers significantly less computational resources, which are essential for on-device applications. Thorough exploration of this idea was done and I am convinced this is a good alternative to regular attention. All the rest of the paper was to incorporate this new method in different tasks in different architectures using or not using attention, and seeing the differences in terms of computational resources. Well thought experiments and results. Again, cannot be verified unless code is shared.
assistant
SUMMARY: A new similarity function replacing the dot product of key and query in attention modules - instead take a shared-weighted sum Good paper, sound theory, very clear explanations. Literature review was sufficient to explain the problem and underlying theory. Results look convincing, although they cannot be verified unless code is shared. Reasonable direction of exploration - there are several possible similarity functions, this paper explores one of them that offers significantly less computational resources, which are essential for on-device applications. Thorough exploration of this idea was done and I am convinced this is a good alternative to regular attention. All the rest of the paper was to incorporate this new method in different tasks in different architectures using or not using attention, and seeing the differences in terms of computational resources. Well thought experiments and results. Again, cannot be verified unless code is shared.