I have the following situation, I’m trying to train a Unet Learner using fastai’s Library. My data is stored as float16 tensor saved by using torch.save and loaded via a custom load function. In fastai, you create a Learner object, and then you call Learn.fit() to train your model. My memory usage is linearly going up during training to a point where I run out of memory. One interesting thing is that my memory usage is reset between epochs, and so the problem is coming from an epoch of training. Basically fastai iters through a pytorch dataloader and does its stuff on top of that.
My Learner item has a learn.data.train_dl.dl attribute which is a torch.utils.data.dataloader.DataLoader. When simply iterating through this dataloader with
> for xb, yb in learn.data.train_dl.dl:
>     pass

All my memory is used in a few minutes. I read on the forums that this could be coming from an issue when using multiple workers to load the files. I tried setting num_workers to 0 but this didn’t fix my memory issue.
I would really like to be able to train my model, do you have any idea on how to fix that bug, or how to do a hack (like how to replace the for x, y in dl with something else) to go around this issue?
For more details on how i’m loading the data, I’m using a fastai custom class TensorImageList(ImageList) where I just overrides the open(self, fn) method with :
class TensorImageList(ImageList):  
    def open(self, fn):
        return torch.load(fn, map_location='cpu').type(torch.float)

I’ve also opened a post on fastai forum discussing this issue : https://forums.fast.ai/t/learn-fit-one-cycle-makes-me-run-out-of-memory-on-cpu-while-i-train-on-gpu/45428 12