Update:
After investigation, I can see that the problem is due to my gradients not updating properly. In the case where my loss fails to decrease, the gradients are zero on every training iteration:
In the failure cases, printing:
loss = ALPHA*nll_loss(output, target)  # nll_loss is torch.nn.NLLLoss with weights
loss.register_hook(lambda grad: print(grad))

...

# perform update
loss.backward()
optimizer.step()
print(model.conv2.weight.grad)
print(model.conv1.weight.grad)
print(model.lstm.weight_ih_l1.grad)
print(model.lstm.weight_ih_l0.grad)

gives the output:
tensor(1., device='cuda:0')
tensor([[[[0., 0., 0., 0.]]]], device='cuda:0')
tensor([[[[0., 0., 0., 0.]],

         [[0., 0., 0., 0.]]]], device='cuda:0')
tensor([[0., 0., 0.,  ..., 0., 0., 0.],
        [0., 0., 0.,  ..., 0., 0., 0.],
        [0., 0., 0.,  ..., 0., 0., 0.],
        ...,
        [0., 0., 0.,  ..., 0., 0., 0.],
        [0., 0., 0.,  ..., 0., 0., 0.],
        [0., 0., 0.,  ..., 0., 0., 0.]], device='cuda:0')
tensor([[0., 0., 0.,  ..., 0., 0., 0.],
        [0., 0., 0.,  ..., 0., 0., 0.],
        [0., 0., 0.,  ..., 0., 0., 0.],
        ...,
        [0., 0., 0.,  ..., 0., 0., 0.],
        [0., 0., 0.,  ..., 0., 0., 0.],
        [0., 0., 0.,  ..., 0., 0., 0.]], device='cuda:0')

I’m not sure why this is happening some times and not others.