Thanks @ptrblck for the quick reply!
Can you explain how to use a summed loss?
I think that in my case I just average all the losses:
for epoch in range(num_epochs):
    print('Epoch {}/{}'.format(epoch, num_epochs - 1))
    print('-' * 10)

    for phase in ['train', 'val']:
        if phase == 'train':
            model.train()
        else:
            model.eval()

        running_loss = 0.0
        successes = 0.0
        total = 0.0

        for inputs, labels in tqdm(dataloaders[phase]):
            inputs = inputs.to(device)
            labels = labels.to(device)
            
            # Set gradients to zero
            optimizer.zero_grad()

            with torch.set_grad_enabled(phase == 'train'):
                outputs = model(inputs)

                # Counting L1 loss
                loss =  criterion(outputs, labels.float())

I tried MSELoss with the same results.
I also considered nn.CrossEntropyLoss but it will apply the same penalty to an error of 7 (instead of 8) than an error of 1 (instead of 8). I agree it may work for this example, but it won’t work if I scale the difficulty of the task.
Thanks!