@ptrblck and @Volker_Siegel I just found the culprit. It seems the evaluating of this happens after I evaluate it:
phi.eval()
inTrainTodevice = inTrain.to(device) # GPU
print("Memory Allocated before Evaluating " + str(torch.cuda.memory_allocated(device)/100000 ) + " MGB")
yPredictThisLevel = phi(inTrainTodevice[:,:indim]) # GPU
print("Memory Allocated after Evaluating " + str(torch.cuda.memory_allocated(device)/100000 ) + " MGB")

The memory allocation jumps like this:
Memory Allocated after dividing of batches 0.43008 MGB
Memory Allocated after Training 1.58208 MGB
Memory Allocated before Evaluating 33.03936 MGB
Memory Allocated after Evaluating 6083.32288 MGB

But I don’t understand why this is the case, since is just a simple evaluation. Do you have any ideas why this would be the case?