Solved by ptrblck in post #8 
              
              
                In your approach you are only rescaling the final output activation of the base vgg model, while it seems that multiple activation maps are scaled and summed to the vgg loss in the paper: 

We define the VGG loss based on the ReLU activation layers of the pre-trained 19 layer VGG network described i…