@ptrblck I discovered the source of the issue!
I’ve had a memory leak that has plagued me while using an LSTM, and to fix the issue, my solution was to use copy.copy() (from the Python standard lib) on tensors in a few places. But in using copy.copy(), it appears to have disrupted the end-to-end nature of the network, preventing proper backpropagation. When I removed copy.copy(), I found that backpropagation works now.
But I still have the memory leak problem and can’t complete a single batch if I’m not copy()ing, so this isn’t a perfect solution. Do you have a suggestion for an alternative solution to the memory leak that doesn’t break backpropagation?
A simplified version of my code is in the following gist:

  
      gist.github.com
  
  
    https://gist.github.com/mhr/2b214985c1bafdbba971590878c83fe3 56
either_backprop_or_leak.py
def find_indices(condition, list_):
    return [i for i, x in enumerate(list_) if condition(x)]

class Network(nn.Module):
    def __init__(self, vocab_dim, embed_dim, input_dim, hidden_dim=512):
        self.embedding = nn.Embedding(vocab_dim, embed_dim)
        self.rnn = nn.LSTM(embed_dim+input_dim,
                           hidden_dim,
                           batch_first=True)
        self.out = nn.Linear(hidden_dim, vocab_dim)
This file has been truncated. show original




  
  
    
    
  
  


In particular, pay attention to lines 49-52. This is where I’m calling copy.copy().
Thank you again for your help.