Robust Text Classifier on Test-Time Budgets
In this paper, we design a generic framework for learning a robust text classification model that achieves accuracy comparable to standard full models under test-time budget constraints. We take a different approach from existing methods and learn to dynamically delete a large fraction of unimportant words by a low-complexity selector such that the high-complexity classifier only needs to process a small fraction of important words. In addition, we propose a new data aggregation method to train the classifier, allowing it to make accurate predictions even on fragmented sequence of words. Our end-to-end method achieves state-of-the-art performance while its computational complexity scales linearly with the small fraction of important words in the whole corpus. Besides, a single deep neural network classifier trained by our framework can be dynamically tuned to different budget levels at inference time.
 The authors consider the setting of training a RNN-based text classification where there is a resource restriction on test-time prediction (e.g., time). Conceptually, the approach is essentially a masking mechanism to reduce the number of words/phrases/sentences used in the prediction at test time followed by a classifier trained to handle these ‘missing’ components. This robustness is achieved by a data aggregation approach that trains on several masked variants determined by the budget level to generalize when encountered new masked instances. This work is most similar to finding ‘rationales’ in text (e.g., [Lei, Barzilay & Jakkola; EMNLP16]) except the selector is much simpler (specifically, using ‘shallow’ learners) and is compensated for with the robust classifier component. Experiments are conducted on four datasets, demonstrating varying levels of performance improvements over baseline and competing methods in terms of accuracy and test-time speedup — followed by more in-depth experiments on multi-aspect sentiment and IMDB datasets, resulting in several more detailed observations. From a high-level perspective, reducing resource requirements specifically at test-time is ostensibly useful in practice as many applications would likely be willing to trade-off some accuracy for {faster performance, lower power requirements, reduced communication bandwidth}, etc. Furthermore, the approach described is sensible and results in some promising empirical findings. My primary concerns with this specific paper are two-fold: (1) the experiments seem mixed and somewhat difficult to interpret in terns of what I would expect to work on a new dataset (2) the tradeoffs aren’t well calibrated in the sense that I don’t know how to set them easily (as everything is performed retrospectively, even if tuned on a dev set, it doesn’t seem trivial to set tradeoffs). Some of this was considered with additional empirical results, but these also didn’t have sufficient clarity nor discussion for me to really understand the recommendation. Accordingly, as is, this seems like a promising idea with solid preliminary evidence of proposed configurations ostensibly working on multiple datasets, but neither a strong theory nor empirical understanding of the dynamics of the proposed methods. Evaluating the paper along the requested dimensions: = Quality: The number of datasets are sufficient and the results are convincing for some configurations. Thus, the authors have established that the method is practical and is promising. However, there isn’t an in-depth study of when the proposed method works, under what configurations, and discussion on how to push this forward. One way to make this more transparent would be to include some curves wrt Table 1 such that I can observe the performance/speedup tradeoffs (is it monotonic?). A good start, but seems incomplete to be really convincing. (4/10) = Clarity: The writing is fairly clear and well-contextualized wrt existing work. I do think that the discussion could be improved overall in terns of better telling the reader what they should be focusing on in the empirical results. My one specific issue was I still don’t understand what is in the parentheses of {LSTM-jump, skim-RNN} in Table 1. Maybe it is in the text, but I really couldn’t decode (and I can understand the semantics of parentheses in other parts of the table). (4/10) = Originality: I haven’t seen anything specifically like this, even if it does draw on very closely related methods. However, I think putting them together required good knowledge from ML and NLP communities, and thus wasn’t trivial and the basic idea is clever. (6/10) = Significance: The underlying idea is interesting and could be significant if further developed, both in terms of better understanding the dynamics and providing better guidance regarding how to do this well in practice. However, just based on the content of the paper, I might play with this, but don’t see it as a solution to problems in low-resource settings, mixed-computation networks (like the cloud example in 4.2). Interesting, but needs more development to be widely adopted. (5/10) === Pros === + an interesting idea that pragmatically builds on existing work to provide a practical solution + experiments conducted on several datasets with some interesting results === Cons === - empirical results mixed and little clarity provide on why - other than speedup, doesn’t consider other settings in experiments (e.g., power, compression) - little theoretical development - nothing said about training time difference (even if not the point) Overall, I like the direction as reducing amortizing computation by focusing on adding resources to training time to reduce during testing time (when most of the volume of predictions should occur). However, I think the idea needs to be further developed before being published at a top-tier venue.
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
Robust Text Classifier on Test-Time Budgets
In this paper, we design a generic framework for learning a robust text classification model that achieves accuracy comparable to standard full models under test-time budget constraints. We take a different approach from existing methods and learn to dynamically delete a large fraction of unimportant words by a low-complexity selector such that the high-complexity classifier only needs to process a small fraction of important words. In addition, we propose a new data aggregation method to train the classifier, allowing it to make accurate predictions even on fragmented sequence of words. Our end-to-end method achieves state-of-the-art performance while its computational complexity scales linearly with the small fraction of important words in the whole corpus. Besides, a single deep neural network classifier trained by our framework can be dynamically tuned to different budget levels at inference time.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



 The authors consider the setting of training a RNN-based text classification where there is a resource restriction on test-time prediction (e.g., time). Conceptually, the approach is essentially a masking mechanism to reduce the number of words/phrases/sentences used in the prediction at test time followed by a classifier trained to handle these ‘missing’ components. This robustness is achieved by a data aggregation approach that trains on several masked variants determined by the budget level to generalize when encountered new masked instances. This work is most similar to finding ‘rationales’ in text (e.g., [Lei, Barzilay & Jakkola; EMNLP16]) except the selector is much simpler (specifically, using ‘shallow’ learners) and is compensated for with the robust classifier component. Experiments are conducted on four datasets, demonstrating varying levels of performance improvements over baseline and competing methods in terms of accuracy and test-time speedup — followed by more in-depth experiments on multi-aspect sentiment and IMDB datasets, resulting in several more detailed observations. From a high-level perspective, reducing resource requirements specifically at test-time is ostensibly useful in practice as many applications would likely be willing to trade-off some accuracy for {faster performance, lower power requirements, reduced communication bandwidth}, etc. Furthermore, the approach described is sensible and results in some promising empirical findings. My primary concerns with this specific paper are two-fold: (1) the experiments seem mixed and somewhat difficult to interpret in terns of what I would expect to work on a new dataset (2) the tradeoffs aren’t well calibrated in the sense that I don’t know how to set them easily (as everything is performed retrospectively, even if tuned on a dev set, it doesn’t seem trivial to set tradeoffs). Some of this was considered with additional empirical results, but these also didn’t have sufficient clarity nor discussion for me to really understand the recommendation. Accordingly, as is, this seems like a promising idea with solid preliminary evidence of proposed configurations ostensibly working on multiple datasets, but neither a strong theory nor empirical understanding of the dynamics of the proposed methods. Evaluating the paper along the requested dimensions: = Quality: The number of datasets are sufficient and the results are convincing for some configurations. Thus, the authors have established that the method is practical and is promising. However, there isn’t an in-depth study of when the proposed method works, under what configurations, and discussion on how to push this forward. One way to make this more transparent would be to include some curves wrt Table 1 such that I can observe the performance/speedup tradeoffs (is it monotonic?). A good start, but seems incomplete to be really convincing. (4/10) = Clarity: The writing is fairly clear and well-contextualized wrt existing work. I do think that the discussion could be improved overall in terns of better telling the reader what they should be focusing on in the empirical results. My one specific issue was I still don’t understand what is in the parentheses of {LSTM-jump, skim-RNN} in Table 1. Maybe it is in the text, but I really couldn’t decode (and I can understand the semantics of parentheses in other parts of the table). (4/10) = Originality: I haven’t seen anything specifically like this, even if it does draw on very closely related methods. However, I think putting them together required good knowledge from ML and NLP communities, and thus wasn’t trivial and the basic idea is clever. (6/10) = Significance: The underlying idea is interesting and could be significant if further developed, both in terms of better understanding the dynamics and providing better guidance regarding how to do this well in practice. However, just based on the content of the paper, I might play with this, but don’t see it as a solution to problems in low-resource settings, mixed-computation networks (like the cloud example in 4.2). Interesting, but needs more development to be widely adopted. (5/10) === Pros === + an interesting idea that pragmatically builds on existing work to provide a practical solution + experiments conducted on several datasets with some interesting results === Cons === - empirical results mixed and little clarity provide on why - other than speedup, doesn’t consider other settings in experiments (e.g., power, compression) - little theoretical development - nothing said about training time difference (even if not the point) Overall, I like the direction as reducing amortizing computation by focusing on adding resources to training time to reduce during testing time (when most of the volume of predictions should occur). However, I think the idea needs to be further developed before being published at a top-tier venue.
assistant
 The authors consider the setting of training a RNN-based text classification where there is a resource restriction on test-time prediction (e.g., time). Conceptually, the approach is essentially a masking mechanism to reduce the number of words/phrases/sentences used in the prediction at test time followed by a classifier trained to handle these ‘missing’ components. This robustness is achieved by a data aggregation approach that trains on several masked variants determined by the budget level to generalize when encountered new masked instances. This work is most similar to finding ‘rationales’ in text (e.g., [Lei, Barzilay & Jakkola; EMNLP16]) except the selector is much simpler (specifically, using ‘shallow’ learners) and is compensated for with the robust classifier component. Experiments are conducted on four datasets, demonstrating varying levels of performance improvements over baseline and competing methods in terms of accuracy and test-time speedup — followed by more in-depth experiments on multi-aspect sentiment and IMDB datasets, resulting in several more detailed observations. From a high-level perspective, reducing resource requirements specifically at test-time is ostensibly useful in practice as many applications would likely be willing to trade-off some accuracy for {faster performance, lower power requirements, reduced communication bandwidth}, etc. Furthermore, the approach described is sensible and results in some promising empirical findings. My primary concerns with this specific paper are two-fold: (1) the experiments seem mixed and somewhat difficult to interpret in terns of what I would expect to work on a new dataset (2) the tradeoffs aren’t well calibrated in the sense that I don’t know how to set them easily (as everything is performed retrospectively, even if tuned on a dev set, it doesn’t seem trivial to set tradeoffs). Some of this was considered with additional empirical results, but these also didn’t have sufficient clarity nor discussion for me to really understand the recommendation. Accordingly, as is, this seems like a promising idea with solid preliminary evidence of proposed configurations ostensibly working on multiple datasets, but neither a strong theory nor empirical understanding of the dynamics of the proposed methods. Evaluating the paper along the requested dimensions: = Quality: The number of datasets are sufficient and the results are convincing for some configurations. Thus, the authors have established that the method is practical and is promising. However, there isn’t an in-depth study of when the proposed method works, under what configurations, and discussion on how to push this forward. One way to make this more transparent would be to include some curves wrt Table 1 such that I can observe the performance/speedup tradeoffs (is it monotonic?). A good start, but seems incomplete to be really convincing. (4/10) = Clarity: The writing is fairly clear and well-contextualized wrt existing work. I do think that the discussion could be improved overall in terns of better telling the reader what they should be focusing on in the empirical results. My one specific issue was I still don’t understand what is in the parentheses of {LSTM-jump, skim-RNN} in Table 1. Maybe it is in the text, but I really couldn’t decode (and I can understand the semantics of parentheses in other parts of the table). (4/10) = Originality: I haven’t seen anything specifically like this, even if it does draw on very closely related methods. However, I think putting them together required good knowledge from ML and NLP communities, and thus wasn’t trivial and the basic idea is clever. (6/10) = Significance: The underlying idea is interesting and could be significant if further developed, both in terms of better understanding the dynamics and providing better guidance regarding how to do this well in practice. However, just based on the content of the paper, I might play with this, but don’t see it as a solution to problems in low-resource settings, mixed-computation networks (like the cloud example in 4.2). Interesting, but needs more development to be widely adopted. (5/10) === Pros === + an interesting idea that pragmatically builds on existing work to provide a practical solution + experiments conducted on several datasets with some interesting results === Cons === - empirical results mixed and little clarity provide on why - other than speedup, doesn’t consider other settings in experiments (e.g., power, compression) - little theoretical development - nothing said about training time difference (even if not the point) Overall, I like the direction as reducing amortizing computation by focusing on adding resources to training time to reduce during testing time (when most of the volume of predictions should occur). However, I think the idea needs to be further developed before being published at a top-tier venue.