query embedding vectors are also tied. LSH self attention uses the locality sensitive hashing mechanism proposed in
Practical and Optimal LSH for Angular Distance to assign each of the tied key
query embedding vectors to one of config.num_buckets possible buckets. The premise is that the more "similar"
key query embedding vectors (in terms of cosine similarity) are to each other, the more likely they are assigned to
the same bucket.
The accuracy of the LSH mechanism can be improved by increasing config.num_hashes or directly the argument
num_hashes of the forward function so that the output of the LSH self attention better approximates the output
of the "normal" full self attention. The buckets are then sorted and chunked into query key embedding vector chunks
each of length config.lsh_chunk_length. For each chunk, the query embedding vectors attend to its key vectors
