You can just save the loss while training / validating your models.
val_loss = 0
for batch_idx, (data, target) in enumerate(val_loader):
    data = Variable(data, volatile=True)
    target = Variable(target, volatile=True)

    output = model(data)

    loss = criterion(output, target)
    
    val_loss += loss.data.cpu().numpy()

val_loss /= len(val_loader)

Then append it to a list / np.array for each epoch and plot it with matplotlib.