Solved by ptrblck in post #2 
              
              
                I would use the validation loss and would not use the test loss at all (neither for the scheduler, nor early stopping etc., as I would consider it a data leak). 
Once your training is finished using the training and validation datasets, you would use the test dataset once and deploy the model.