How can i compute seq2seq loss using mask?
In TensorFlow, i can do this as below. <SCODE># targets is an int64 tensor of shape (batch_size, padded_length) which contains word indices.    
# masks is a tensor of shape (batch_size, padded_length) which contains 0 or 1 (0 if pad otherwise 1).

outputs = decoder(...)  # unnormalized scores of shape (batch_size, padded_length, vocab_size) 
outputs = tf.reshape(outputs, (-1, vocab_size))
targets = tf.reshape(targets, (-1))
losses = tf.nn.sparse_softmax_cross_entropy_loss(outputs, targets)    # loss of shape (batch_size*padded_length)

masks = tf.reshape(masks, (-1))
loss = losses * masks    
<ECODE> <SCODE>criterion = nn.CrossEntropyLoss()
outputs = decoder(features, inputs)   # (batch_size, padded_length, vocab_size)
loss = criterion(outputs.view(-1, vocab_size), targets.view(-1))  # this gives a scalar not tensor
<ECODE> How can i solve this problem?
yunjey
jekbradbury
<SCODE>cap = CocoCaptions(root = './data/train2014resized',
                   annFile = './data/annotations/captions_train2014.json',
                   vocab = vocab,
                   transform=transform,
                   target_transform=transforms.ToTensor())

data_loader = torch.utils.data.DataLoader(
    cap, batch_size=16, shuffle=True, num_workers=2)

for i, (images, input_seqs, target_seqs, masks) in enumerate(data_loader):
    # images: a tensor of shape (batch_size, 3, 256, 256).
    # input_seqs, target_seqs, masks: tensors of shape (batch_size, padded_length).<ECODE>
yunjey
<SCODE>import torch.nn.utils.rnn as rnn_utils  
lstm = nn.LSTM(in_size, hidden_size, bidirectional, num_layers)
packed = rnn_utils.pack_padded_sequence(padded, lengths)
packed_out, packed_hidden = lstm(packed)
unpacked, unpacked_len = rnn_utils.pad_packed_sequence(packed_out)<ECODE>
jekbradbury
supakjk
apaszke
supakjk
jekbradbury
Could you let me know how to deal with the following example case? Assume that I have an input, where maximum sequence length is 5, minibatch size is 4, and there are 3 possible labels. idxs has effective lengths of sequences in the input. <SCODE>input = Variable(torch.randn(5,4,3))
idxs = [5,3,3,2]
target = Variable(torch.LongTensor(5,4))
# assume the target labels are assigned corresponding to the input.
<ECODE> Thanks!
supakjk
pranav
apaszke
Then the cross_entropy loss could calculated with <SCODE>def nll_loss(outputs, targets):
    return torch.gather(outputs, 1, targets.view(-1,1))
<ECODE>
Response777
Can you help me understand this torch.gather I have used the same tensors  with similar shape but this line torch.gather(outputs, 1, targets.view(-1,1)) gives me error.
vijendra_rana
Thanks for sharing the code. I test the nll_loss function and get: RuntimeError: Input tensor must have same dimensions as output tensor at /b/wheel/pytorch-src/torch/lib/TH/generic/THTensorMath.c:424 It seems that the input and index arguments of gather function should have same dimensions.
volpato30
currently I use this way to compute masked loss for seq2seq model: <SCODE>'''
target_sentences: [batch_size, max_length] int64
batch_target_len_array: [batch_size] int64, each element is the valid length of the corresponding target sentence. 
'''
for step in range(max_length):
    decoder_output, decoder_hidden = decoder(
        decoder_input, decoder_hidden, encoder_outputs)
    if np.sum(batch_target_len_array > step) == 0:
        break
    mask_vector = torch.from_numpy((batch_target_len_array > step).astype(np.int32)).byte()
    index_vector = Variable(torch.masked_select(torch.arange(0, batch_size), mask_vector).long()).cuda()
    valid_output = torch.index_select(decoder_output, 0, index_vector)
    valid_target = torch.index_select(target_sentences[:, step], 0, index_vector)

    step_loss = criterion(valid_output, valid_target)
    loss += step_loss
    decoder_input = target_sentences[:, step]  # Teacher forcing<ECODE>
volpato30
<SCODE>loss_function = nn.CrossEntropyLoss(ignore_index=0)
<ECODE> since i have zero padded targets.
cbaziotis
I have tried, doesn’t work in my case. Could you post your code with a bit more details?
Madotto_Andrea
It is clear. I’ll try! The newest version 0.3 updated the cost function with a new Parameter “reduce”, and it’s easy to mask the loss
Littleye
Can you explain how to mask the loss now?
himat
With the parameter “reduce”, we can get the loss per batch element, but how can we use the mask on it? For example, if I have a minibatch whose valid sequence lengths is [3,1,2], and with “reduce” we can get three loss values [L1, L2, L3]. But we need to mask the last two values in calculating L2 and last one value in calculating L3. How could it be achieved? ------------------------------edited---------------------------- Oh I see, we only need to use ignore_index=0 parameter in loss(), but don’t need the “reduce” parameter to achieve this. Is it right?
jpzhou