Hi,
This error occurs when i put the model for training.
The model architecture is initialized to an object model . This is where the memory fills up. then i run model.cuda() -> this point it tries to allocate that 10.24 GiB, results in cuda mem error.
Here is the story:
Training for 1st time: works fine. used gpu 2,3 (takes 25min for one epoch) -> process quit
Training for 2nd time: works fine. switched to gpu 2 (takes 17min for one epoch) -> process quit because i wanted to save best model. So made changes to check val loss and save.
Training for 3rd time: above described cuda error.
I used the same code to run on a different system worked like a charm.