Hadn’t read about this. Cool stuff!

Every 10k training steps, they remake their batches by computing the cosine similarity of every pair of docs, and then greedily adding source and target docs to batches such that the pairwise sum of cosine similarities increases the most.

You seem to imply that this is not an expensive operation, but it sounds very expensive: calculate vector for doc, cos sim between all data points greedily. Isn’t that super computationally expensive?