Update:
The problem came from the validation calculation actually. I do a validation loss calculation each 50 epochs and my validation dataset size is 72 which is too large. That is why I have an error " CUDA out of memory" while finishing the epoch 49 and starting the epoch 50.
I will change the code in that way and hopefully it will work !