When I implemented the ResNet architecture, I too bumped into the same problem of out of memory execution. I realized that my GPU was getting populated since I was having a batch size of 64. So I reduced my batch size to 4 and it worked fine. This is just one such scenario.