Update:
I did another test where I kept the number of training iterations small than previously. This time the RMSE during validation (i.e. 4.54 m) is closer to the expected value from pre-training (4.48 m) but still changes from epoch to epoch.
In a second test I did not call loss.backward() during training and the validation RMSE still changes from epoch to epoch.
Three possibilities

The validation dataloader is somehow messing up the data.

forward() itself alters the parameters.
The optimizer is somehow altering my depthnet parameters even when they are not added to the optimizer and .detach() is called.

I think 3. is just not possible but I am completely clueless right now…
Will further investigate
Update:
This time I completely removed the optimizer from the equation and the RMSE still changes.
Therefore I think the optimizer is doing fine.