Hi，thanks for your help. sorry to bother you again.
A new problem came in my model, could you tell me to print something in the code to find where the bug exactly was?
the error was :
CuDNN error: CUDNN_STATUS_EXECUTION_FAILED

屏幕快照 2019-04-12 下午10.04.01.png2096×694 167 KB

I was using dataparallel, when it went to compute reward,
    def forward(self, targets, inputs, corpus):
        fake_reply, word_probabilities, hiddens = self.G.sample(inputs, targets, TF=0)
        num_samples = 3
        rewards = self.G.monte_carlo(self.D, inputs, fake_reply, hiddens, num_samples, corpus).detach()
    def monte_carlo(self, dis, context, reply, hiddens, num_samples, corpus):
        
        # Initialize sample
        batch_size = reply.size(0)
        vocab_size = self.decoder.output_size
        encoder_output, _ = self.encoder(context)
        rewards = torch.zeros(self.max_len, num_samples, batch_size)
        function = F.log_softmax
        reply = reply
        for t in range(self.max_len):
            for n in range(num_samples):
                samples = reply.clone()
                hidden = hiddens[t]
                output = reply[:, t]
                # Pass through decoder and sample from resulting vocab distribution
                for next_t in range(t + 1, self.max_len):
                    decoder_output, hidden, step_attn = self.decoder.forward_step(output.reshape(-1, 1).long(), hidden, encoder_output, function=function)
    def forward_step(self, input_var, hidden, encoder_outputs, function):
        batch_size = input_var.size(0)
        output_size = input_var.size(1)
        embedded = self.embedding(input_var)
        embedded = self.input_dropout(embedded)
        self.rnn.flatten_parameters()
        output, hidden = self.rnn(embedded, hidden)

Here is other relevant code with forward_step:
self.embedding = nn.Embedding(self.output_size, self.hidden_size)
rnn_cell='gru'
self.rnn = self.rnn_cell(hidden_size, hidden_size, n_layers, batch_first=True, dropout=dropout_p)

should I do something with the output and hidden in monte_carlo  before I call the forward_step  ?
or should print them to see where they are?
I tried printing embedded and embedded in forward_step function,they are distributed in devices(1,2,3).Then I got losted and have no way to debug,
Beg for your help!Thanks a lot!!