I have a very large buffer of images (~1 million). They are all different. I can’t fit all of the images on the GPU, so they’re all on the CPU originally. Then, I sample from the buffer, do some things with the sampled images, and repeat. Something like (pseudocode):
buffer = torch.ones(10**6, 32, 32, 3) # images are 32x32x3
for step in range(10**8):
  images = sample(buffer, 256).to('cuda')
  update_model_with_images(images)

There’s no way to predict which images will be sampled, so I can’t pre-fetch them. When I profile my code using torch.utils.bottleneck, I find that almost all of the runtime is spent on moving tensors from CPU to GPU. This is frustrating. How can I increase performance without fitting the entire buffer on my GPU?