I found out that I forgot to use an activation function (now relu()) for the last two fully connected layers. Now it works out quite well! I am still thankful for any other tips or comments on how to improve regression performance using a CNN!