Depending on the speed of this operation, you might calculate the similarity matrix inside the DataLoader loop, while the GPU is busy with the model training.
However, I’m not sure what method you are using and how slow/fast it is on the CPU and GPU.
Could you post the applied method and the relative time it’s using in a training iteration?