The train and validation code looks fine to me.

Why are you using loss_value.backward(retain_graph=True) and not just loss_value.backward()?
Any particular reason for using L1 loss instead of any other loss (eg: CrossEntropyLoss)?