The problem was that I was the approach taken in mean_teacher, which is to have each image saved individually as a file, which is loaded each time the dataloader is called. That was of course horribly inefficient for small networks. The problem was solved by moving to torchvision as suggested.