I investigated further, and the issue seems to be that after 31 training steps the model would always get an extra long sequence and run out of memory, just as @ngimel initially surmised.  This took me longer to realize than it should have, in part because pytorch doesn’t reliably give an error message that says “out of memory” when this happens.  Depending on the details of the model and input, it can give other error messages like
torch.backends.cudnn.CuDNNError: 8: b'CUDNN_STATUS_EXECUTION_FAILED'

and
RuntimeError: invalid argument 2: out of range at /home/grant/pubrepos/pytorch/torch/lib/THC/generic/THCTensor.c:23