what are you loading here? images or some other format?
Someone recently reported when loading Tiff, the python library they are using to load had memory leaks.
Also, are you using a custom Dataset class or using one from torchvision / torchtext?