Solved by Shi_Heng in post #12 
              
              
                Hi, This OOM exception comes from the python api implement of conv2d_weight actually. 
In backprop weight calculation, the output gradients need to be expanded with output channel times. When default cudnn implement this with data prefetch block and block (not allocate more memory), python api uses …