For each approach I used 20 000 epochs. I used random seeds to have the same output every time.
I noticed that the first ~2000  epochs the values of R en C (parameters I added to the model to fit) are the same using both approaches (1. torch.mean(torch.square(output - labels)) and 2. criterion = nn.MSELoss()) . After these epochs, the values start to differ. Here I show you the training timelines of Parameter C:
 Using 1. torch.mean(torch.square(output - labels))
 Using 2. criterion = nn.MSELoss()
Again, the only thing I changed was the method I used to describe the loss, although they should be (and are) the same. Both losses give identical every epoch, but when using method 1. to learn on, the parameters are getting other values then when the model is learned on method 2. (as shown in the figures). Therefore, other losses are found between both methods.
I am just wondering why this difference shows up.  Since the losses are the same during training, but different if it trains on the other loss-function, I thought it might be the case that the backward function works differently for both approaches.