Solved by PauleB in post #4 
              
              
                Update: 
The problem came from the validation calculation actually. I do a validation loss calculation each 50 epochs and my validation dataset size is 72 which is too large. That is why I have an error " CUDA out of memory" while finishing the epoch 49 and starting the epoch 50. 
I will change the …