Oh I see the issue. Okay maybe like this:
...
loss.backward()
for w1, w2 in zip(lstm1.parameters(), lstm2.parameters()):
  w1.grad.data.add_(w2.grad.data)
  w2.grad = None
optim.step()
for w1, w2 in zip(lstm1.parameters(), lstm2.parameters()):
  w2.data.copy_(w1.data)

Might be slow though