I have also tried to use a pool of CUDA streams to avoid having to recreate them at each timestep, and I have removed the torch.cuda.synchronize()s which don’t seem to be useful.
But still, the training speed decreases linearly with the number of heads