I believe the issue is actually due to the for loop in the attention class. The following lines specifically:
   # Create variable to store attention energies
    attn_energies = Variable(torch.zeros(this_batch_size, max_len)) # B x S

    if USE_CUDA:
        attn_energies = attn_energies.cuda()

    # For each batch of encoder outputs
    for b in range(this_batch_size):
        # Calculate energy for each encoder output
        for i in range(max_len):
            attn_energies[b, i] = self.score(hidden[:, b], encoder_outputs[i, b].unsqueeze(0))

it is faster to do this calculation using torch functions. Something like attn_weights = torch.bmm(encoder_outputs, decoder_output). Though you’ll have to reshape encoder_outputs and decoder_output.