Transformers, Pytorch, Machine Learning, Deep Dives, Data Science.
model.vocab_size), y[:, 1:].contiguous().view(-1)) loss.backward() optimizer.step() losses += loss.item() preds = logits.argmax(dim=-1) masked_pred = preds * (y[:, 1:]!=PAD_IDX) accuracy = (masked_pred == y[:, 1:]).float().mean() acc += accuracy.item() history_loss.append(loss.item())