Solved by ptrblck in post #4 
              
              
                I don’t know what might be causing the issue as I’m able to overfit random data using your model and this code: 
N = 64
data = torch.randn(N, 10, 1)
device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')

model = Discriminator(hidden_size=32, target_seq_len=1, num_classes=5)
optimizer…