Anchor & Transform: Learning Sparse Representations of Discrete Objects
Learning continuous representations of discrete objects such as text, users, and items lies at the heart of many applications including text and user modeling. Unfortunately, traditional methods that embed all objects do not scale to large vocabulary sizes and embedding dimensions. In this paper, we propose a general method, Anchor & Transform (ANT) that learns sparse representations of discrete objects by jointly learning a small set of anchor embeddings and a sparse transformation from anchor objects to all objects. ANT is scalable, flexible, end-to-end trainable, and allows the user to easily incorporate domain knowledge about object relationships (e.g. WordNet, co-occurrence, item clusters). ANT also recovers several task-specific baselines under certain structural assumptions on the anchors and transformation matrices. On text classification and language modeling benchmarks, ANT demonstrates stronger performance with fewer parameters as compared to existing vocabulary selection and embedding compression baselines.
The paper describes a *layer* that aims at producing embeddings for discrete objects by using fewer parameters than classical embeddings layers. Indeed, the model proposes, instead of learning an embedding matrix of size VxN, to learn a matrix of embeddings of anchors (AxN) and a transformation matrix (VxA) such that the embedding of any object can be found by multiplying A with T. On top of that, they propose different regularization techniques to improve the quality of the learned embeddings, and particularly a proximal gradient method over a L1 normalization on T to reduce the number of parameters. They propose also different ways to initialize A and also a method for incorporating a priori information (e.g knowledge) into the model. They evaluate this model on different tasks: text classification and language modeling and show that they can achieve good performance while using fewer parameters than Sota methods. First of all, the paper is well written, and the description is very detailed and understandable. It was a pleasure to read such a paper! One point which is unclear is the interest of using such a method, and more precisely in which cases, this method can be useful. Indeed, the overall number of parameters of ANT is AxN + VxA (N being the size of the embeddings, A the number of anchors and V the size of the vocabulary) while classical methods are VxN parameters. Said otherwise, we need to have V<N to really have less parameters to train in the model -- knowing that classical embeddings spaces size is usually between 256 and 1024, it means that we have to target a task where the number of anchors is quite low. I agree that the sparsity term on T is here to encourage to decrease the number of parameters but first, the same sparsity could be applied on the original VxN embedding matrix, and also, even if, at the end, the T matrix is sparse, during learning one has to maintain a large matrix in memory. I would like the authors to discuss more on this point which is crucial? Particularly, I am not sure to understand what the #Emb value is in the table (AxN + AxV or just AxN), and how to compare the models. (There is a discussion in Section 3, but the argumentation does not explain why having so many parameters at train time is not a problem). Also, since this is the crucial point in the paper, I would be interested in having a discussion about the use of neural models compression techniques after learning that could also *do the job* (even if they are not trained end-to-end). One other remark concerns the different *components* added into the model (e.g sparsity, orthogonality, Relu...). It is difficult to measure the interest of each of them, and I would recommend the authors to provide an ablation study to make the effect of the different choices more understandable by the reader. The notion of anchors also is misleading since it gives the impression that the A matrix will store embeddings for particular objects, while there is no constraint of that type. Each line of the A matrix is an embedding, but this embedding is not associated with one of the objects seen at train time (no direct mapping from anchors to words in the vocabulary). This has to be made more clear at the beginning of the paper. Concerning the initialization of A by K-means, it assumes that the space of objects has a particular metric. The authors say that this metric can come from a pretrained embedding space, but in that case, the problem in the number of parameters (which is the main justification of this work) is invalid (i.e if you already have an embedding matrix, then just let us fine-tune it). Could you clarify ? The fact that the method would allow incorporating knowledge is certainly the most interesting point. The way it is done has to be better explained (I do not understand why positive pairs are taken into account by not enforcing sparsity on T at this particular point, the way negative pairs are handled seem more natural) The paper is interesting and proposes a new simple model that could be used to keep good performance while reducing the number of parameters of the final model. Discussions have to be added to discuss the relevance of the approach since it still needs a large number of parameters at train time, and the role of each component could be studied more in depth.
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
Anchor & Transform: Learning Sparse Representations of Discrete Objects
Learning continuous representations of discrete objects such as text, users, and items lies at the heart of many applications including text and user modeling. Unfortunately, traditional methods that embed all objects do not scale to large vocabulary sizes and embedding dimensions. In this paper, we propose a general method, Anchor & Transform (ANT) that learns sparse representations of discrete objects by jointly learning a small set of anchor embeddings and a sparse transformation from anchor objects to all objects. ANT is scalable, flexible, end-to-end trainable, and allows the user to easily incorporate domain knowledge about object relationships (e.g. WordNet, co-occurrence, item clusters). ANT also recovers several task-specific baselines under certain structural assumptions on the anchors and transformation matrices. On text classification and language modeling benchmarks, ANT demonstrates stronger performance with fewer parameters as compared to existing vocabulary selection and embedding compression baselines.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



The paper describes a *layer* that aims at producing embeddings for discrete objects by using fewer parameters than classical embeddings layers. Indeed, the model proposes, instead of learning an embedding matrix of size VxN, to learn a matrix of embeddings of anchors (AxN) and a transformation matrix (VxA) such that the embedding of any object can be found by multiplying A with T. On top of that, they propose different regularization techniques to improve the quality of the learned embeddings, and particularly a proximal gradient method over a L1 normalization on T to reduce the number of parameters. They propose also different ways to initialize A and also a method for incorporating a priori information (e.g knowledge) into the model. They evaluate this model on different tasks: text classification and language modeling and show that they can achieve good performance while using fewer parameters than Sota methods. First of all, the paper is well written, and the description is very detailed and understandable. It was a pleasure to read such a paper! One point which is unclear is the interest of using such a method, and more precisely in which cases, this method can be useful. Indeed, the overall number of parameters of ANT is AxN + VxA (N being the size of the embeddings, A the number of anchors and V the size of the vocabulary) while classical methods are VxN parameters. Said otherwise, we need to have V<N to really have less parameters to train in the model -- knowing that classical embeddings spaces size is usually between 256 and 1024, it means that we have to target a task where the number of anchors is quite low. I agree that the sparsity term on T is here to encourage to decrease the number of parameters but first, the same sparsity could be applied on the original VxN embedding matrix, and also, even if, at the end, the T matrix is sparse, during learning one has to maintain a large matrix in memory. I would like the authors to discuss more on this point which is crucial? Particularly, I am not sure to understand what the #Emb value is in the table (AxN + AxV or just AxN), and how to compare the models. (There is a discussion in Section 3, but the argumentation does not explain why having so many parameters at train time is not a problem). Also, since this is the crucial point in the paper, I would be interested in having a discussion about the use of neural models compression techniques after learning that could also *do the job* (even if they are not trained end-to-end). One other remark concerns the different *components* added into the model (e.g sparsity, orthogonality, Relu...). It is difficult to measure the interest of each of them, and I would recommend the authors to provide an ablation study to make the effect of the different choices more understandable by the reader. The notion of anchors also is misleading since it gives the impression that the A matrix will store embeddings for particular objects, while there is no constraint of that type. Each line of the A matrix is an embedding, but this embedding is not associated with one of the objects seen at train time (no direct mapping from anchors to words in the vocabulary). This has to be made more clear at the beginning of the paper. Concerning the initialization of A by K-means, it assumes that the space of objects has a particular metric. The authors say that this metric can come from a pretrained embedding space, but in that case, the problem in the number of parameters (which is the main justification of this work) is invalid (i.e if you already have an embedding matrix, then just let us fine-tune it). Could you clarify ? The fact that the method would allow incorporating knowledge is certainly the most interesting point. The way it is done has to be better explained (I do not understand why positive pairs are taken into account by not enforcing sparsity on T at this particular point, the way negative pairs are handled seem more natural) The paper is interesting and proposes a new simple model that could be used to keep good performance while reducing the number of parameters of the final model. Discussions have to be added to discuss the relevance of the approach since it still needs a large number of parameters at train time, and the role of each component could be studied more in depth.
assistant
The paper describes a *layer* that aims at producing embeddings for discrete objects by using fewer parameters than classical embeddings layers. Indeed, the model proposes, instead of learning an embedding matrix of size VxN, to learn a matrix of embeddings of anchors (AxN) and a transformation matrix (VxA) such that the embedding of any object can be found by multiplying A with T. On top of that, they propose different regularization techniques to improve the quality of the learned embeddings, and particularly a proximal gradient method over a L1 normalization on T to reduce the number of parameters. They propose also different ways to initialize A and also a method for incorporating a priori information (e.g knowledge) into the model. They evaluate this model on different tasks: text classification and language modeling and show that they can achieve good performance while using fewer parameters than Sota methods. First of all, the paper is well written, and the description is very detailed and understandable. It was a pleasure to read such a paper! One point which is unclear is the interest of using such a method, and more precisely in which cases, this method can be useful. Indeed, the overall number of parameters of ANT is AxN + VxA (N being the size of the embeddings, A the number of anchors and V the size of the vocabulary) while classical methods are VxN parameters. Said otherwise, we need to have V<N to really have less parameters to train in the model -- knowing that classical embeddings spaces size is usually between 256 and 1024, it means that we have to target a task where the number of anchors is quite low. I agree that the sparsity term on T is here to encourage to decrease the number of parameters but first, the same sparsity could be applied on the original VxN embedding matrix, and also, even if, at the end, the T matrix is sparse, during learning one has to maintain a large matrix in memory. I would like the authors to discuss more on this point which is crucial? Particularly, I am not sure to understand what the #Emb value is in the table (AxN + AxV or just AxN), and how to compare the models. (There is a discussion in Section 3, but the argumentation does not explain why having so many parameters at train time is not a problem). Also, since this is the crucial point in the paper, I would be interested in having a discussion about the use of neural models compression techniques after learning that could also *do the job* (even if they are not trained end-to-end). One other remark concerns the different *components* added into the model (e.g sparsity, orthogonality, Relu...). It is difficult to measure the interest of each of them, and I would recommend the authors to provide an ablation study to make the effect of the different choices more understandable by the reader. The notion of anchors also is misleading since it gives the impression that the A matrix will store embeddings for particular objects, while there is no constraint of that type. Each line of the A matrix is an embedding, but this embedding is not associated with one of the objects seen at train time (no direct mapping from anchors to words in the vocabulary). This has to be made more clear at the beginning of the paper. Concerning the initialization of A by K-means, it assumes that the space of objects has a particular metric. The authors say that this metric can come from a pretrained embedding space, but in that case, the problem in the number of parameters (which is the main justification of this work) is invalid (i.e if you already have an embedding matrix, then just let us fine-tune it). Could you clarify ? The fact that the method would allow incorporating knowledge is certainly the most interesting point. The way it is done has to be better explained (I do not understand why positive pairs are taken into account by not enforcing sparsity on T at this particular point, the way negative pairs are handled seem more natural) The paper is interesting and proposes a new simple model that could be used to keep good performance while reducing the number of parameters of the final model. Discussions have to be added to discuss the relevance of the approach since it still needs a large number of parameters at train time, and the role of each component could be studied more in depth.