Hi there,
I’m new to pytorch and trying to train a GNN with my limited knowledge…However, I met a problem with CUDA running out of memory. My dataset contains graphs with variable sizes.
By printing out the memory allocated by Cuda and the memory reserved by Cuda, it seems if I don’t use a sampler in DataLoader, the memory reserved stays unchanged, but if I use a sampler (with replacement = True), the memory reserved is different, and keep increasing. I’m guessing that if I unluckily sampled a bunch of large graphs needs a lot more memory. However, why after each iteration, those memory are not freed? Is this because I used the sampler in a wrong way?
I’m currently using it by:
loader = GraphDataLoader(ds_train, batch_size=batch_size, drop_last=True, sampler=sampler)
Here is the two memory usage comparison:
Without sampler:

Screen Shot 2020-04-10 at 2.06.32 PM356×724 15.9 KB

With sampler:

Screen Shot 2020-04-10 at 2.04.34 PM410×1042 21.6 KB

Thanks for your time looking at this!