So far I end up with following way.
First I modify torch.nn.utils.rnn.pad_sequence and let it returns the lengths of each sample.
def pad_sequence(sequences, batch_first=False, padding_value=0):
    trailing_dims = sequences[0].size()[1:]
    max_len = max([s.size(0) for s in sequences])
    if batch_first:
        out_dims = (len(sequences), max_len) + trailing_dims
    else:
        out_dims = (max_len, len(sequences)) + trailing_dims
    out_tensor = sequences[0].new(*out_dims).fill_(padding_value)
    lengths = sequences[0].new_zeros(len(sequences),dtype=torch.long)
    for i, tensor in enumerate(sequences):
        lengths[i] = tensor.size(0)
        if batch_first:
            out_tensor[i, :lengths[i], ...] = tensor
        else:
            out_tensor[:lengths[i], i, ...] = tensor

    return out_tensor, lengths

Then I pass these two variables into the module. You can wrap out_tensor and lengths into a dictionary. Inside module, I for-loop batch-wise, and for each sample or mini-batch with the same sequence length, I for-loop sequence-wise. Finally, I concatenate the output of each sample to evaluate the loss.
Be aware, if you want to do this on multi-GPUs with nn.DataParallel,  you should use pad_sequence() with batch_first = True.