Solved by SimonW in post #2 
              
              
                Yes, it will be higher on one GPU, but only slightly. The amount of memory needed to store inputs and coalcse is usually small compared to the amount needed for autograd.