We extend the idea of word pieces in natural language models to machine learning tasks on opaque ids. This is achieved by applying hash functions to map each id to multiple hash tokens in a much smaller space, similarly to a Bloom filter. We show that by applying a multi-layer Transformer to these Bloom filter digests, we are able to obtain models with high accuracy. They outperform models of a similar size without hashing and, to a large degree, models of a much larger size trained using sampled softmax with the same computational budget. Our key observation is that it is important to use a multi-layer Transformer for Bloom filter digests to remove ambiguity in the hashed input. We believe this provides an alternative method to solving problems with large vocabulary size.
The paper proposes to use codes based on multiple hashing functions to reduce the input and output dimensions of transformer networks. The novelty of the idea is mostly to integrate these codes with transformers. The authors present experiments on two tasks showing that keeping overall model size fixed (i..e, number of parameters), transformers with shorter (but denser) binary codes achieve better performances than standard transformers using one-hot encodings. The gain mostly comes from using larger embedding sizes and larger intermediate layers. 
While the technical contribution is limited, because most of the principles are already known or straightforward, the main contribution of the paper is to show that random hash functions are sufficient to create significantly shorter codes and maintain good performances. Such codes provide more freedom in terms of where to put model capacity (larger embeddings, more transformer layers, etc.) which may be useful in applications where most of the model parameters are in embedding matrices. 
The value of such a paper resides mostly in the experimental study. On the bright side, the experiments present in sufficient details the impact of the various hyper parameters and the new trade-offs model size/performance that can be achieved. On the other hand, the experiments are carried out on non-standard tasks without previously published baselines, and it is unclear why. Since the method is applicable to any problem involving natural language data (and more generally categorical values, such as knowledge base completion), I would have expected experiments on tasks with a well-defined state-of-the-art. This makes the experiments in the paper look more like "proofs of concept", and they are less convincing than they should be.
detailed comments:
- The paper really is *not* about bloom filters (Bloom filters are data structures that represent sets and efficiently answer membership queries). It is about using codes of identifiers of lower dimension than one-hot encoding. This idea has been used in multi class classification setting (i.e., for the output layer) since (at least) Dietterich & Bakiri (1995) "Solving Multiclass Learning Problems via Error-Correcting Output Codes" (with more insights on what makes a good code for prediction problems). The authors borrow from Bloom filter the way to create the codes using random hash functions, but the analogy stops here. 
- following the comment above, and assuming I understood correctly: There is an originality on the paper compared to other works that use binary codes/bloom filters: In the current paper, the authors actually predict the result of individual hashing functions. This is different from predicting the binary encoding that results from using the "or" of one-hot encodings generated by several hash functions, as would be done in approaches (truely) based on Bloom filters. For instance, if there are m hash functions taking values in {1, ..., P}, an approach based on Bloom filters would predict a binary output of dimension P, while here there are m multiclass problems with P classes (IIUC). This difference from previous work may be significant in practice.
- I found the description of the link prediction task (section 3.1) rather cryptic: 
* "from each page, we take a random contiguous segment of entities". If I understand clearly, the text is filtered out and only links are kept (?). Links are replaced by the entity id they point to. What happens to the entity the page is about? Is it added at the start?
* 'For evaluation, we hold out one random entity from a random segment on a test page. ": what does "holding out" mean? From my understanding, it means replaced by the [MASK] token, but it could also mean removed altogether from the input sequence.
* the task is to predict masked links in sequences of links with the surrounding text filtered out. Does that correspond to any real-life prediction problem (i don't see which one)? Is this "task" intended to serve as unsupervised pre-training of embeddings? If yes, maybe the authors might say so and give example applications.
- For the natural language task: 
* ", then apply a Bloom filter to reduce the vocabulary size." -> From my understanding, there is no bloom filter here. If I understand, what is done is to represent unigrams and bigrams by their bloom digest to reduce the input dimension.
- "We hold out 10% random entity pages for testing. " -> is there a validation set? How do you choose the hyper parameters?
minor comments:
- "Since the Wikipedia site usually removes duplicates of links on each page, the distribution of pages is rather long tail." -> I wouldn't be surprised if the distribution was long tailed even without this specific policy
- I found the formalization/notation more confusing than helping because it is not really thorough (there is no distinction between sets and sequences, "1[\eta_j(t_i)] ... where 1[.] is the indicator function" -> what is the "indicator function" of a number?)