ptrblck:

This shouldn’t be the case and cudnn should discard all algorithms which could cause an OOM error. Have you seen an OOM using cudnn.benchmark ?


Actually my real training code is a complicated version of the posted one. It’s not just a cross entropy loss but a distributed manner in which every process computes hand-crafted loss and gradients. Several tensor are used as intermediate variables. Will cudnn take these into consideration when choosing algorithms?