Does your validation loop yield the OOM directly during the first iteration or after a few iterations?
Could you post a (small) code snippet, which shows how you are using the training and validation loop?