If the out of memory error is raised in the second epoch, you might be storing unnecessary tensors (or the whole computation graph) in some variables.
E.g.
avg_bleu_score+=avg_bleu
avg_gleu_score+=avg_gleu

might attach the whole computation graph, if avg_* is still attached to it (you can check if by printing the .grad_fn or to be safe just detach() these tensors).
Also, to save memory, you could wrap the validation loop into a with torch.no_grad() block.
To use nn.DataParallel you should wrap the model in nn.DataParallel and make sure the input tensors can be split in dim0, so that each chunk is passed to the corresponding device.
This tutorial 3 will give you more information.