Thanks.  Possibly this is the issue.  I am not calling detach, but I just throw away the final hidden state, and then run the RNN on a fresh sequence (starting with a fixed initial hidden state that currently has requires_grad set to False, although I don’t think that should matter).  So I don’t think I need to call detach.  Also, if I am just not freeing memory properly, it is confusing why it always happens on the 32nd iteration even when changing parameters like batch size and sequence length, and it is confusing why nvidia-smi shows constant memory usage from the first iteration right until the crash.