The randperm call is quite expensive. Especially since you only need 1000 indices.
You could use torch.randnint, if duplicates are OK. Otherwise maybe you could permute once outside the loop and then use a “sliding” window getting your mini batch size.
This would affect both the CPU and GPU code.
Were you able to run my small timing script?