Sounds to me like you might be accidentally using the following anti-pattern:
for input, answer in batches:
    optimizer.zero_grad()
    output = model(input)
    loss_var = loss_func(output, answer)
    loss_var.backward()
    optimizer.step()
    del loss_var # missing this!

Without the del the loss_var is still holding a reference to gradients all the way through the second iteration of the loop, including during the calculation of loss_func. Python can’t garbage collect the old value until after the new has been computed and assigned to loss_var!
If it’s not clear what I mean, try this:
loss_var = None
for input, answer in batches:
    optimizer.zero_grad()
    output = model(input)
    print loss_var
    loss_var = loss_func(output, answer)
    loss_var.backward()
    optimizer.step()

Good luck!