If cudnn.benchmark = True, it will automatically find the proper algorithms based on the GPUs, and may choose different convolution algorithms for K80 and TitanX, which have different memory consumptions. Maybe try to set it to False and see if they are the same.