Solved by ptrblck in post #2 
              
              
                The memory usage could be different based on e.g. the picked cudnn algorithms, i.e. a deterministic one could use less memory but might also be slower.