I have a possible solution for a similar problem. I also store a large variable in the dataloader object, and I came across the same memory problem. There is one bug on PyTorch: if you set num_worker < physical cpu kernels, it works fine; otherwise, pytorch seems to replicate the dataloader object in memory, which lead to memory leak.
In short, there is a BUG in PyTorch, try to set num_worker < physical cpu kernels.