Kronecker Attention Networks
Attention operators have been applied on both 1-D data like texts and higher-order data such as images and videos. Use of attention operators on high-order data requires flattening of the spatial or spatial-temporal dimensions into a vector, which is assumed to follow a multivariate normal distribution. This not only incurs excessive requirements on computational resources, but also fails to preserve structures in data. In this work, we propose to avoid flattening by developing Kronecker attention operators (KAOs) that operate on high-order tensor data directly. KAOs lead to dramatic reductions in computational resources. Moreover, we analyze KAOs theoretically from a probabilistic perspective and point out that KAOs assume the data follow matrix-variate normal distributions. Experimental results show that KAOs reduce the amount of required computational resources by a factor of hundreds, with larger factors for higher-dimensional and higher-order data. Results also show that networks with KAOs outperform models without attention, while achieving competitive performance as those with original attention operators.
In this paper, the authors propose to reduce the memory and computation complexity of attention networks applied to 2D data (images) by using, in the attention operator, the mean over the rows and columns of images instead of its vectorized version. The paper is relatively well written, based on intuitive ideas and including clear figures. Several experimental results are reported but, in my opinion, they are not convincing enough. Below, a list of issues is presented: - I think the term “Kronecker” in the name of the model is not well justified or explained. I don’t see a Kronecker product of matrices in the attention operator. Maybe, only in the case of KAO_{QKV} the outer sum could be written as a Kronecker product of some matrices justifying the name. I think this is not a big issue but it is a little bit annoying to use a term that is not naturally introduced to the reader. - In my opinion, the main issue is that taking the mean over columns and rows as the only information to construct the attention operator seems not able to capture all relevant information about the underlying dataset. I think there is not a theory that justifies this approach. For example, it could be possible that other linear or nonlinear combination of rows and columns provides better attention operators. - It is not clear how the theoretical results (Theorem 1) helps to support the proposed method. Maybe the authors could better explain the implications of their theoretical results. I found the introduction of the method clear but rather disconnected from the theoretical sections 3.3-3.5. - Experimental results are not convincing. I cannot agree with the author’s claim that “KANets significantly outperform the previous state-of-the-art compact models”. Below I make comments to each of the presented experimental results. o The experimental results about memory and time used by regular attention operator compared with KAOs is obvious and not relevant because their theoretical complexities are well known. o In Table 2, it is shown that the proposed method is better than other non-attentional networks. I think this result is not relevant. On the other side, in Table 3, they compare the proposed method against two other attentional networks and show that the proposed method is not better than previous methods in terms of performance. The only advantage seems to be a slight reduction in number of operations which is marginal. o In the Ablation Studies (Table 4), the method is not compared with other classical attention network. They only compared against MobileNet. I wonder how the performance results would be if classical attention networks were used having the same level of number of parameters. o The results on Image Segmentation are not relevant because the proposed method is not the best one.
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
Kronecker Attention Networks
Attention operators have been applied on both 1-D data like texts and higher-order data such as images and videos. Use of attention operators on high-order data requires flattening of the spatial or spatial-temporal dimensions into a vector, which is assumed to follow a multivariate normal distribution. This not only incurs excessive requirements on computational resources, but also fails to preserve structures in data. In this work, we propose to avoid flattening by developing Kronecker attention operators (KAOs) that operate on high-order tensor data directly. KAOs lead to dramatic reductions in computational resources. Moreover, we analyze KAOs theoretically from a probabilistic perspective and point out that KAOs assume the data follow matrix-variate normal distributions. Experimental results show that KAOs reduce the amount of required computational resources by a factor of hundreds, with larger factors for higher-dimensional and higher-order data. Results also show that networks with KAOs outperform models without attention, while achieving competitive performance as those with original attention operators.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



In this paper, the authors propose to reduce the memory and computation complexity of attention networks applied to 2D data (images) by using, in the attention operator, the mean over the rows and columns of images instead of its vectorized version. The paper is relatively well written, based on intuitive ideas and including clear figures. Several experimental results are reported but, in my opinion, they are not convincing enough. Below, a list of issues is presented: - I think the term “Kronecker” in the name of the model is not well justified or explained. I don’t see a Kronecker product of matrices in the attention operator. Maybe, only in the case of KAO_{QKV} the outer sum could be written as a Kronecker product of some matrices justifying the name. I think this is not a big issue but it is a little bit annoying to use a term that is not naturally introduced to the reader. - In my opinion, the main issue is that taking the mean over columns and rows as the only information to construct the attention operator seems not able to capture all relevant information about the underlying dataset. I think there is not a theory that justifies this approach. For example, it could be possible that other linear or nonlinear combination of rows and columns provides better attention operators. - It is not clear how the theoretical results (Theorem 1) helps to support the proposed method. Maybe the authors could better explain the implications of their theoretical results. I found the introduction of the method clear but rather disconnected from the theoretical sections 3.3-3.5. - Experimental results are not convincing. I cannot agree with the author’s claim that “KANets significantly outperform the previous state-of-the-art compact models”. Below I make comments to each of the presented experimental results. o The experimental results about memory and time used by regular attention operator compared with KAOs is obvious and not relevant because their theoretical complexities are well known. o In Table 2, it is shown that the proposed method is better than other non-attentional networks. I think this result is not relevant. On the other side, in Table 3, they compare the proposed method against two other attentional networks and show that the proposed method is not better than previous methods in terms of performance. The only advantage seems to be a slight reduction in number of operations which is marginal. o In the Ablation Studies (Table 4), the method is not compared with other classical attention network. They only compared against MobileNet. I wonder how the performance results would be if classical attention networks were used having the same level of number of parameters. o The results on Image Segmentation are not relevant because the proposed method is not the best one.
assistant
In this paper, the authors propose to reduce the memory and computation complexity of attention networks applied to 2D data (images) by using, in the attention operator, the mean over the rows and columns of images instead of its vectorized version. The paper is relatively well written, based on intuitive ideas and including clear figures. Several experimental results are reported but, in my opinion, they are not convincing enough. Below, a list of issues is presented: - I think the term “Kronecker” in the name of the model is not well justified or explained. I don’t see a Kronecker product of matrices in the attention operator. Maybe, only in the case of KAO_{QKV} the outer sum could be written as a Kronecker product of some matrices justifying the name. I think this is not a big issue but it is a little bit annoying to use a term that is not naturally introduced to the reader. - In my opinion, the main issue is that taking the mean over columns and rows as the only information to construct the attention operator seems not able to capture all relevant information about the underlying dataset. I think there is not a theory that justifies this approach. For example, it could be possible that other linear or nonlinear combination of rows and columns provides better attention operators. - It is not clear how the theoretical results (Theorem 1) helps to support the proposed method. Maybe the authors could better explain the implications of their theoretical results. I found the introduction of the method clear but rather disconnected from the theoretical sections 3.3-3.5. - Experimental results are not convincing. I cannot agree with the author’s claim that “KANets significantly outperform the previous state-of-the-art compact models”. Below I make comments to each of the presented experimental results. o The experimental results about memory and time used by regular attention operator compared with KAOs is obvious and not relevant because their theoretical complexities are well known. o In Table 2, it is shown that the proposed method is better than other non-attentional networks. I think this result is not relevant. On the other side, in Table 3, they compare the proposed method against two other attentional networks and show that the proposed method is not better than previous methods in terms of performance. The only advantage seems to be a slight reduction in number of operations which is marginal. o In the Ablation Studies (Table 4), the method is not compared with other classical attention network. They only compared against MobileNet. I wonder how the performance results would be if classical attention networks were used having the same level of number of parameters. o The results on Image Segmentation are not relevant because the proposed method is not the best one.